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EVALUATING BEHAVIORALLY ORIENTED AVIATION MAINTENANCE 
RESOURCE MANAGEMENT (MRM) TRAINING AND 
PROGRAMS:METHODS, RESULTS, CONCLUSIONS 1 

James C. Taylor, Ph.D. 

& Robert L. Thomas, Ph.D. 

School of Engineering 

Santa Clara University 
Santa Clara, CA 95053-0590 

SUMMARY 

Assessment of the impact of Aviation Resource Management Programs on 
aviation culture and performance has compelled a considerable body of research (Taylor 
& Robertson, 1995; Taylor, 1998; Taylor & Patankar, 2001). In recent years new 
methods have been applied to the problem of maintenance error precipitated by factors 
such as the need for self-assessment of communication and trust. The present study - 
2002 — is an extension of that past work. 

This research project was designed as the conclusion of a larger effort to help 
understand, evaluate and validate the impact of Maintenance Resource Management (MRM) 
training programs, and other MRM interventions on participant attitudes, opinions, behaviors, 
and ultimately on enhanced safety performance. It includes research and development of 
evaluation methodology as well as examination of psychological constructs and correlates of 
maintainer performance. 

In particular, during 2002, three issues were addressed. First the evaluation of two 
(independent & different) MRM programs for changing behaviors was undertaken. In one 
case we were able to further apply the approach to measuring written communication 
developed during 2001 (Taylor, 2002; Taylor & Thomas, 2003). Second the MRM/TOQ 
surveys were made available for completion on the internet. The responses from these “on- 
line surveys” were automatically linked to a results calculator (like the one developed and 
described in Taylor, 2002) to aid industry users in analyzing and evaluating their local survey 
data on the internet. Third the main trends and themes from our research about MRM 
programs over the past dozen years were reviewed. 


1 Acknowledgements : We wish to thank Dr. Barbara Kanki, NASA-ARC, for her generous support and 
encouragement for our research over the past five-plus years. Our colleague Dr. Manoj Patankar, Associate 
Professor, Parks College of Engineering and Aviation, St. Louis University, has been most helpful in 
providing advice and ideas throughout the course of our research. Mr. Blain Hamon brought his talents as 
“webmaster” to bear to produce the web-based survey and results calculator, which until this past year was 
only a dream. And finally Ms. Shelley Shiltz,, our undergraduate research assistant since 2000, continued 
to provide high quality support during the year. 
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I. Evaluation of Behaviorally Oriented MRM Training. 

A. Data Collected 

1. Pre-and Post-training survey comparisons to test the effect of formal training 
input focused on behavior changes. 

2. Post-training and Follow-up survey comparisons to test the effect of 
subsequent training-inspired behavioral program implementation 

3. Field Interviews & Observation 

4. Written Performance Data 

5. Cases illustrating use of the new behaviors 


The Survey Measure: The "Maintenance Resource Management Technical 
Operations Questionnaire" (MRM/TOQ). 

The MRM/TOQ developed for the present study is a further modification of a 
survey for which development began in 1991 (Taylor, 2000b). The MRM/TOQ 
questionnaire is a self-report measure of attitudes and opinions that are related 
(conceptually or empirically) to human factors (HF) and safety training in maintenance 
and maintenance support functions. Respondents are asked to express their degree of 
agreement to some 17 statements. A five-point agreement scale is used for each question 
or statement. Four Likert-type scales are constructed through the combination of the 
results of these statements. The development, as well as the reliability and validity of 
these scales are reported in Taylor & Thomas (in press). The meaning and utility of these 
scales are summarized as follows. 

“Trust Supervisor’s Safety Practices.” A high score on this first scale reflects the 
degree of trust respondents have for their supervisor or manager on safety related matters. 
Survey questions that comprise this scale probe for how much the respondent feels they 
can approach management without fear of punishment, backlash or inaction (especially 
with safety issues and suggestions). 

Previous research has shown that aircraft mechanics (Aviation Maintenance 
Technicians [AMTs]) are more apt to be loners than other vocations (Taylor, 1999); and 
in the U.S. they tend to value teamwork and HF training less than their counterparts in 
other countries (Taylor & Patankar, 1999). With low trust levels of AMTs for their 
supervisors ranging between one-quarter and two-thirds, in a sample of five maintenance 
organizations (Patankar, Taylor, & Goglia, 2002), the problem of trust is a real one. The 
implications of this problem are multi-faceted and long-term. For one thing, the incident 
and error investigation programs such as ASAP (FAA, 1997) and MED A (Allen & 
Rankin, 1995) require threshold levels of mutual trust between management and AMTs 
that may not be met by many carriers. For another thing, because too many MRM 
training programs suffer from a lack of visible management support and encouragement 
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(Taylor, 1998, 2000c; Taylor & Patankar, 2001), we believe that their long-term 
acceptance also requires mutual trust. Thirdly, only increasing this mutual trust will offer 
an avenue to resolving the industry-wide paradox called the speed-accuracy trade-off, or 
“SATO” (Drury & Gramopadhye, 1991), in which AMTs emphasize quality of 
maintenance and their managers promote production speed. 

“Value Coworker Trust & Communication. ” A high score on this second scale, 
expresses a high value for trusting one’s coworkers’ as well as communicating with them 
in meetings and discussions. Taylor & Thomas (in press) report that “Value Coworker 
Trust & Communication” is sensitive to the differences between the departments. AMTs 
in base-maintenance hangars are assigned to work together on complex jobs lasting as 
much as a week, while AMTs in flight line tend to be assigned to work alone on much 
shorter jobs. These conditions may well engender greatest value for collaboration among 
the base-hangar AMTs and the lesser value for this attribute on the flight line. 

Attempts to change organizational culture and to improve trust among 
maintenance personnel, although not a direct focus of change in most MRM programs, is 
clearly an important byproduct of programs intended to open channels of communication 
and improve collaboration. The two measures of trust, above, provide a baseline and 
initial assessment in order to concentrate on efforts to improve trust first before investing 
heavily in subsequent programs requiring high trust to succeed. In ongoing programs the 
trust measures should be most useful to track and evaluate such programs and would 
offer a comparison across companies as well as comparing the same company over time. 

“Value of Assertiveness.” The assertiveness scale has been successfully adapted 
from earlier research (Taylor, 2000b). A resultant high score on this scale emphasizes the 
value of candor and openness among maintenance personnel. Openness and honesty 
have proven to be important in maintenance HF programs (Taylor & Robertson, 1995). 
Valuing assertiveness has consistently shown positive relationships with subsequent 
safety outcomes (Taylor, Robertson & Choi, 1997; Taylor & Patankar, 2001). 

“Value of Recognizing Stress Effects.” This scale too, has been adapted from 
earlier work in evaluating aviation HF programs (Taylor, 2000b). A high score means a 
high value placed on the consideration of stressors at work and the utility of 
compensating for them. Though not related to the theme of human communication (or 
indirectly to interpersonal relations), this factor proves to be an important concept for 
maintenance professionalism and is central to the curriculum of many HF training 
programs (Taylor & Patankar, 2001). Understanding and managing stress are primarily 
passive coping activities; and improvement following the training correlated to safety 
performance improvements has been established in previous studies in terms of the 
marked increase in appreciation of stress management after training revealing strong 
correlations with low rates of injury and aircraft damage (c/, Taylor, 1998). 

Other items in The MRM/TOO. The survey also contains several demographic 
questions, as well as three items that measure enthusiasm and support for the program. 
Additionally, open-response questions yield data regarding how the program was used, 
further intentions to use the program, and respondents’ thoughts about how to improve 
the program. 
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1 . Pre and post-training effects. The MRM/TOQ is used to measure employee 
attitudes, opinions and background data immediately before MRM training. 
Immediately after the training these items are asked again, together with three 
fixed response items measuring enthusiasm about the training, and three open- 
ended questions asking about respondents (Rs) expected changes in behavior, as 
well as highlights and needed improvement for the training. 

2. Follow-up effects . The MRM/TOQ is also used to measure after a program has 
been in place for a period of time. It measures the same things as the post- 
training survey, above, but also asks about what changes R has seen in 
him/herself and in the organization in general. 

3. Field interviews . Months after the program commences, we visit the worksite and 
we ask Rs to comment about what they remember about the training, what 
changes R has made, what changes R has seen in others, and how the program 
might be improved. 

4. Written performance data . In the present study, the only performance data 
collected were in Company 1, as reported below. The measures used were the 
quality of written discrepancy log entries. 

5. Case studies . As an adjunct to the performance data collected in Co. 1, episodes 
of using their decision making process were documented as “cases” for feedback 
to the participants, as well as for evidence of errors avoided through the use of the 
process. 
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I. Evaluation of Behaviorally Oriented MRM Training: 

B. Company 1, Corporate Aviation Department 

• Field Interviews & Survey data 

• Written Performance (Discrepancy Reports) Data 

• Cases illustrating use of the new behaviors 

Introduction 

The study in this present section is an extension of past work, but was focused in 
a small corporate aviation facility (eight maintenance technicians). Examination of a 
smaller maintenance facility allowed for observation of an ongoing team with more 
personal connection and interaction than a typical commercial airline. Hence, the sample 
provided by Co. 1 provided an ideal setting for detailed observation of their culture and to 
learn more about the use and impact of a behavioral communication process designed to 
mitigate error. 

The Program 

Implementation of the QP Program 

The initial period: 1995-1998. The behavioral resource management program 
in use by Co. 1 is called QuantumPro ® (QP), a proprietary a program designed by a 
external Consultant 2 . An earlier effort to implement QP had already achieved some 
success in Co. 1 (Patankar & Taylor, 1999). The QP program had initially been 
implemented by the Flight Operations group in 1995, and in 1997-98 had been informally 
introduced into maintenance. 

The reintroduction of OP: 2001. Between 1998 and 2001 there were many 
changes in Co. 1 ’s Aviation Department (including, but not limited to downsizing 
personnel, liquidating one-half of the fleet; then corporate mergers and acquisitions, with 
subsequent rapid department growth requiring rehiring some former employees and 
hiring new ones, together with the purchase of several new aircraft). In 2001 the QP 
Process received added management support and endorsement. 

The current version of the QP program implemented in Co. l’s maintenance 
department was unique to the industry in a number of ways. First it combined one-day 
training sessions with take-home assignments that were spread over a six-month period. 
Second, the program avoided demands to change participant’ attitudes or values and 
appealed directly for changed behaviors. Third, the program focused on avoiding the 
occurrence of errors rather than waiting to understand errors after they had been 
committed. 

The initial training occurred in maintenance during June, 2001. In addition to 
the formal training, follow-up assignments, and repeated use of the MRM/TOQ survey, 
the present researchers became further involved with Co. 1 by engaging in periodic 


2 For more information about QuantumPro and its creators, please refer to Patankar & Taylor, 1999 or 
Patankar, Taylor & Thomas, 2003 
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feedback discussions of our findings and conclusions. The first of these occurred one 
year after the training (including six months of Co. 1 ’s attempts to implement the QP 
process). At that time the members of the maintenance group requested the present 
investigators to interview maintenance personnel about the program and provide them 
with our findings, our conclusions, and our recommendations for next steps. This was 
accomplished in June 2002. Thereafter the present researchers visited the site on a 
weekly basis, until February 2003. During these visits maintenance crewmembers were 
asked to reflect upon their behavior and their experiences with the QP program. This 
research and evaluation process itself no doubt contributed in some part to the outcomes 
described below. 

Elements of QuantumPro 

Briefing and Debriefing. A cornerstone of the QP program is consistency in (a) 
briefing before a maintenance action to identify a shared plan, strengths, weaknesses, and 
other vital information so that team members are in a position to act as back-ups and raise 
alternative solutions, and (b) debriefing after a maintenance task to reflect upon how the 
plan was executed, any further weaknesses, and review the decision-making process. 
These two fundamental components are important because they allow the rest of the team 
to act as a resource on any given individual task. According to the program training 
curriculum, initiation of briefing and debriefing is the responsibility of every member of 
the maintenance team. 


The Concent Alignment Process (CAP). Consistency in briefing and debriefing 
allows for the implementation of a consistent communication process shared and used by 
the entire team. Called the concept alignment process, or CAP, it is the second pillar of 
the QP process. The CAP is a communication protocol for AMTs, as well as pilots, to 
follow not just in briefings and debriefings, but also in their every-day interactions and 
self-reflections. The step-by-step CAP flowchart is displayed in Figure 1. 

The CAP begins with some difference or conflict, whether between two or more 
team members, between flight and technical operations, or between two or more 
competing concepts of an individual technician. Such an example is where several 
AMTs take the same measurement and arrive at conflicting results: some measurements 
are within legal limits and some are not. Using more customary decision models, the 
decision to let the aircraft fly might come down to majority rules, the individual 
experience of the technician, or the department hierarchy (i.e., the supervisor is the last 
word). Using the CAP, however, AMTs in this scenario are forced to validate their own 
concept (i.e., re-measurement, alternative instruments, calibration of own instruments, 
etc.). If the conflict still exists, then at least one extra source must be sought to validate 
each of the competing concepts. Based on the new information, one of the concepts is 
chosen and executed only if each of the parties is satisfied with the validation process. 
Once the plan is executed, the action should then be debriefed to examine the decision- 
making process, and determine what future improvements could be made for future 
situations (e.g., procedure changes, behavior and communication modifications) to 
prevent future occurrences of similar errors. An important aspect of CAP is, if the third 
source of information does not confirm one or another of the misaligned concepts - and if 
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the issue at hand is a safety matter - then the most conservative (i.e., the safest) concept 
is chosen and agreed to by all. 



“FINISH 

# 2 ” 

Prevent ^ 1 

“Latent 
Failures” 


“FINISH 

# 1 ” 

Eliminate 

“Active 

Failure” 


Figure 1. Flowchart of the Concept Alignment Process (CAP) 


The Task Coordinator. A critical component to the program is its intention to 
minimize power distance in the department hierarchy so that team members feel able to 
speak-out or challenge the concept of any other member, even those more experienced or 
in management positions. The coordinator for a given task should be assigned during 
briefings, and is responsible for initiating and driving the CAP, and insisting on 
appropriate validation and follow-up. The task coordinator role was introduced in the 
program as a way to provide an organizational role outside of the hierarchy while using 
the CAP. In the scenario provided in the previous section, imagine that one of the AMTs 
taking the measurement was the department manager. In many cases, challenging the 
concept of the department manager and insisting on validation may be difficult for a 
technician. The task coordinator role diminishes these hierarchical roles and allows those 
with less experience and less organizational authority to insist upon adequate validation. 

Research 

The primary objective of this study was to examine the cultural and behavioral 
impact of the QP maintenance resource management program. The smaller size of the 
subject department and the closeness of the maintenance group allowed for a rich 
examination of the evolution of communication and trust among them. The program as 
implemented was designed to create an atmosphere of team-based decision making and 
high trust in which information is shared, and safer decisions are made based on team 
support. Hence, we expected to see more willingness to raise concerns about processes 
and decisions, more validation of uncertain or conflicting concepts, greater consistency in 
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briefing and debriefing among maintenance and pilot crew members, and procedural and 
continued behavioral changes resulting from successes using the CAP in decisions. 


Methods. 

Data for the current study were collected in three ways: (a) survey administration, 
(b) interviewing and observation over an extended period, and (c) time-series 
examination of archival performance data. All three of these methodologies were 
combined to produce the richest possible understanding of the safety culture in this 
corporate aviation facility. 


Participants . The primary participants in the study were seven AMTs and one 
maintenance supervisor. Other members of the aviation department under study (e.g., 
pilots, department manager) were in some cases involved when their participation could 
lend clarification or background to cases and stories. Co. 1 AMTs varied considerably in 
age, experience with the company, and general aviation experience. These demographics 
are displayed in Table 1. 

Table 1 

Mean Years of Experience for Co. 1 Maintenance Group (N= 8) 



Lowest 

Highest 

Mean 

Other Airline Experience 

3.0 

20.0 

9.50 

College Experience 

2.0 

3.0 

2.50 

Trade School Experience 

2.0 

3.0 

2.33 

Military Experience 

0.0 

5.0 

3.00 

Tech. Ops. Experience 

9.0 

23.0 

14.25 

Age 

31 

60 

47.50 


ll 





Procedure 


Survey administration. AMTs were administered the MRM/TOQ, or Maintenance 
Resource Management Technical Operations Questionnaire (Taylor &Thomas, in press), 
immediately before and after the program’s initial training session, and then at intervals 
thereafter up to 20 months. 

Observation and interviewing. Co.l’s aviation department was also studied 
through regular on-site observations and interviews. This approach required that the 
observers of the culture have access for a relatively long period, and at a level that 
allowed the data collector to learn of the direct experiences and the perspective of the 
participants in the culture. The interviewers/observers became familiar and trusted figures 
who visited the facility on a frequent basis for eight months. Through this level of 
integration with the culture, insights could be derived at a depth not possible through 
survey or less intrusive means. Additionally, interview responses and observations could 
be analyzed with the participants after the data were collected so that frameworks for 
observation could be applied as the field study continued. 

This method of data collection can be considered ethnographic in nature, but with 
the very specific purpose: to assess and evaluate the impact of the aviation resource 
management-training program. Visits were made to the corporate aviation department of 
twice per week between July 2002 and February 2003. Shift meetings and pre-flight and 
post-flight debriefs were observed, and maintenance technicians and the maintenance 
supervisor were interviewed on an individual basis. Content of questions posed in 
interviews ranged from general attitudes and ideas about the program, to inquiries 
regarding specific behavioral practices and cases demonstrating instances where the 
program was used. From participant responses, a casebook (presented below) was 
constructed, which demonstrated the team’s use of the CAP, as well as other aspects of 
the implemented program. 

Examination of archival performance data. Performance data were collected to 
assess the scope and nature of behavior change occurring in coincidence with the QP 
resource management program. Data were collected from aircraft discrepancy logs kept 
in the maintenance archives from January of 1998 to May of 2003. A discrepancy log is 
a record of observations related to aircraft maintenance made by pilots to be addressed by 
maintenance staff. The maintenance staff first hear the discrepancies from a post-flight 
debriefing with the pilots, which they pass on to others during the maintenance shift- 
change meeting, and they are later able to refer to these discrepancies in written form in 
the log. After correcting or addressing the discrepancy, mechanics write a description of 
the work conducted (i.e., corrective action) in the discrepancy log, and then sign their 
entry, thereby accepting responsibility for the work. By examining changes in written 
behavior as evinced through discrepancy log entries, evidence could be obtained that the 
program is having some measurable effect on written communication among pilots and 
mechanics. 
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Results 


Surveys 


The several figures below show changes in levels of the attitudes and enthusiasm 
for Co. 1 maintenance personnel over the 20 month period from June 2001 to January 
2003. The number for responses is small for each time period because the maintenance 
group is small. As a consequence, most of the statistical “F” tests for differences among 
the means, over time, do not reach significance. Statistical significance is shown on each 
graph. As a way of comparing Co. 1 mean scores to the larger industry standard, 
percentile scores are also noted, where appropriate. 

Attitudes 


Trusting Supervisor’s Safety Practices. Figure 2 shows the mean scores for 
trusting the supervisor’s safety practices. There is an overall upward trend in trusting 
supervisor (albeit not strong enough-to reach significance with these small n’s) from the 
pre-training period to 20 months. 

Figure 2 



Co.1 AMT: Trust Supervisor's Safety Practices (F<1, n.s.) 


Pre- Post- 
training training 

6/2001 


6 mo 12 mo 

follow-up follow-up 
11/2001 6/2002 


18 mo 20 mo 
follow-up followup 
11/2002 (1/2003) 


The percentile rank of these means with the industry standard are favorable. 
Although the Co.l pre-training mean ranks slightly below the 50 th percentile, the post 
training rank is above the 60 lh percentile and all of the rest are above the 70 th percentile, 
with 6 and 20 months reaching the 80 th percentile. 

The high six month score represents a hopeful outlook during a time when 
attempts to use QP were few, but the ideas from the training six months earlier were still 
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attractive (representative quote at 6mo: “Great potential for less threatening relationship 
between boss and mechanic, but we’re not there yet”). At 12 months, Trust of supervisor 
drops (but still not even to the “post-training” level) as the group began thinking about 
using CAP in earnest (representative quote at 12 months: “Somebody has to get us to use 
CAP as a team.”). 

Later ( cf . 18 and 20 months in Fig.2), as CAP began to be used and discussed 
more frequently (resulting in cases like the ones summarized in the case book, below), 
trust in supervisor rose and approached the level at six months. Our data collected at the 
1 8 month follow-up reveals that a majority of AMTs are using CAP and plan to continue 
doing so (representative quote: “I feel more comfortable with the program and have no 
problems with the challenge concept 

Value of trusting coworkers. Figure 3 shows the Co. 1 mean scores for this scale 
over the same pre-training to 20 month follow-up. In this case the changing value shows 
a downward (but non-significant) trend as maintenance personnel gained more 
experience with the QP program. 

Figure 3 


Co.1 AMT: Value Coworker Trust & Communication (F=1.66, 

n.s.) 



Pre- Post- 6 mo 12 mo 18 mo 20 mo 
training training follow-up follow-up follow-up followup 
6/2001 11/2001 6/2002 11/2002 (1/2003) 


Comparing these means with the industry standards, the pre-training score is 
ranked at nearly the 80 th percentile, while the post training level is ranked at 70 th 
percentile and subsequent Co.l follow-up surveys are ranked lower, with the last two 
below the 50 th percentile. 

After the 18 month follow-up survey Co. l’s maintenance personnel discussed the 
steady decline in their mean scores, shown in Fig.3, and agreed that because the QP 
process was based on a factual and logical approach to decision making, and was 
beginning to work quite well, they were thus released from simply trusting one another 
that all work decisions had been correct ones. There was no doubt among them that their 
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coworkers were trustworthy (and perhaps even more so than earlier), but the value of this 
construct was less important to them under the QP management process. 

Value of assertive communication. Figure 4 shows Co.l mean scores for 
valuing assertiveness over time. The results in Figure 4 reflect a positive trend in this 
aspect. Co. 1 employees are well above the industry standard for the value they place on 
speaking up. Even before the QP training their percentile ranking was above the 60 th , and 
their post-training , 18 and 20 month scores are ranked nearly at the 80 th percentile. The 
QP process encourages mechanics speaking up and challenging one another and Figure 4 
suggests an improvement over time. In interviews, Co.l’s maintenance personnel 
reported that speaking up was an increasingly valuable way to behave at work. 


Figure 4 


Co.l : Value Assertiveness (F<1 , n.s.) 



Pre-tmg Post-trng 6mo 11/01 12mo6/02 18mo 20mo 1/03 
6/01 6/01 1 1/02 


Value of recognizing stress effects. Figure 5 shows Co.l’s mean scores on the 
scale, “Effect of stress.” This scale measures respondents’ views on the effects of stress 
on decision making. This scale has been shown to be sensitive to training modules on 
stress management in other HF programs (Taylor & Christensen, 1998). However, unlike 
those other approaches to maintenance human factors the QP process does not address 
stress, or stress effects, at all. Since the standard MRM/TOQ survey (Taylor & Thomas, 
in press) was used in the present case however, the scale was included, and as such, it 
acts as a control item - no consistent change in this attitude was expected over time; and 
as Figure 5 shows, none was found. 

The Co.l mean score levels for “Stress effects” compare favorably to industry 
standards before the training (Pre-training ranks at the 67 th percentile), but drops, 
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comparatively speaking in subsequent time periods. With the exception of the 12 month 
follow-up results all after-training results rank at or slightly below the 50% percentile. 


Figure 5 


Co. 1 Value Recognizing Stress Effects (F<1, n.s.) 



Pre- Post- 6 mo 12 mo 18 mo 20 mo 
training training follow-up follow-up follow-up followup 
6/2001 11/2001 6/2002 11/2002 (1/2003) 


Enthusiasm for the QP program 


Figures 6 through 9 show Co.l AMTs’ responses to four questions asking their 
opinions about the program. In all cases these results are positive and compare 
reasonably well with the aviation maintenance industry standard. 
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Figure 6 shows that Co.l employees retain a positive evaluation of the QP effect 
on safety and teamwork during the last three follow-up surveys. Post-training surveys 
rank near the 60 th percentile against all-industry standards, but this ranking drops to 
slightly below the 50 th percentile for subsequent periods. 


Figure 6 


Co.l AMT: This QP program can increase safety & teamwork 

(F<1, ns) 



Post-training 6 mo follow- 12 mo follow- 18 mo follow- 20 mo 
6/2001 up 11/2001 up 6/2002 up 11/2002 followup 

(1/2003) 


I 


17 



Figure 7 shows that Co.l employees’ assessment of the program’s value for 
others may diminish slightly. Post-training surveys rank near the 70 th percentile against 
all-industry standards, but this ranking drops to as low as the 40 th percentile for several of 
the subsequent periods. That is because the industry standard has higher mean scores 
subsequent to post -training scores for this question compared with Co.l. These higher 
scores for the industry are often because of AMTs’ dominant view that their managers 
should participate more in their HF programs. 

Figure 7 


Co.l AMT: This QP program will be useful to others (F<1, ns) 



Post-training 6 mo follow- 12 mo follow- 18 mo follow- 20 mo 
6/2001 up 11/2001 up 6/2002 up 11/2002 followup 

(1/2003) 


During the interviews and observation in Co.l, it was clear that a few of the 
mechanics were not finding the QP process comfortable for them, while the rest took to it 
immediately. As time passed, the number of skeptics diminished, but those who 
remained so, also grew stronger in their reluctance to apply the process. Information 
from interviews suggests that this mixed response is responsible for the results in Figure 
7 - as many in the group were realizing that it might not be so useful for those holdouts. 


Intentions & Behaviors 

A number of Co.l mechanics were feeling that they were changing and would 
continue to change as they applied the QP process 

Figure 8 shows mean scores for the 4-point scale, “this program will change my 
behavior.” The steady (although non-significant) upward trend over time indicates that 
many of the mechanics as individuals are taking the program to heart and intend to 
continue to apply it. Compared with the rest of industry, Co. 1 ’s mean scores on this item 
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are below the 50 th percentile immediately post-training, but their ranking increases over 
time and reaches 60 th percentile in the 18 and 20 month surveys. 


Figure 8 


Co.1 AMT: This program will change my behavior (F<1, n.s.) 



Post-training 6 mo follow- 12 mo 18 mo 20 mo 

6/2001 up 11/2001 follow-up follow-up followiup 

6/2002 11/2002 1/2003 


Some representative quotes at the time of the 18 month survey were: 

• “I have decided to make it my personal goal to ensure that [CAP] debriefs are 
complete and correct. This will be one of my contributions to the team in the 
future. I am still working on correcting and completing debriefs we have 
conducted recently which I feel are still pending, and that management apparently 
is satisfied with.” 

• “I think we as a team have worked through some of our flaws and are more 
careful now to pass on information. Although we still have faults to work on, I 
think we have recognized how important this [QP] is and that there really is no 
such thing as irrelevant information. I personally like to tell everyone everything 
and let them sort it out individually.” 

• “What’s good about QP? The analysis of an event; discussing what went right 
and what went wrong brings out ideas or change in procedures on how to improve 
things so errors, and almost errors can be avoided. That’s having a positive 
outcome no matter what happened during the event.” 


19 




Figure 9 shows that Co.l mechanics see the positive changes increasing over 
time. In fact this positive trend is quite steep and is statistically significant. 


Figure 9 


Co.l AMT: I've seen positive changes for safety since QP was 
introduced (F= 3.71, p<.05) 



12 mo follow-up 18 mo follow-up 20 mo followup 

6/2002 11/2002 (1/2003) 


Open-ended Questions. 

Open-ended questions in the surveys asked Co.l respondents how they intended 
to use the QP information they had learned, and what changes they had made as a result 
of the program. Figures 10 and 1 1 show changes in Co.l mechanics’ responses to these 
two questions, over time. 

Figure 10 shows that immediately following the QP training in June 2001 over 
50% of the mechanics intended to apply the QP processes (including CAP), while another 
20% said they would communicate more or be more a part of the team. 
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It is also seen in Figure 10, that after two periods (6 & 12 months after training) 
where few AMTs expected to use QP any further, the 18 month survey suddenly showed 
a marked increase in which over 80% said they would use QP. 


Figure 10 


Co.1 "How will you apply QP?" 



Communication Interaction/Teamwork Be More Assertive Error Management 

(CAP) 


■ Post-trng D6-mo Ei 12-mo ■ 18-mo 


Few of these AMTs stated that they intended to be more assertive, although we 
know from the results in Figure 4, above, that they increased their value of that behavior 
far beyond the industry standard. 


Figure 11 


Co.1 "What changes have you made?" 



Communication Interaction/Teamwork Be More Assertive Error Management 

(CAP) 


□ 6-mo H 12-mo ■ 18-mo 


The self-reports in Figure 1 1 show that shows QP is the dominant behavior at 12 
& 18 months. 
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Observations and Interviews 


Field Observations 


Briefing and debriefing. Briefings and debriefings were manifest in several ways 
in Co.l’s aviation department. The most formal and consistent is the 15:30 shift meeting 
at which the maintenance supervisor, day shift, and night shift AMTs are present. The 
maintenance supervisor facilitates the day shift reporting of duties, problems and 
solutions, and unfinished tasks to the night shift. The agenda for the next shift is then 
reviewed, as well as forecasts for the days and possibly weeks ahead. Discussion of the 
use of the Quantum-Pro process is a standing if informal item on the daily meeting 
agenda. These daily meetings (five days per week, Monday through Friday) are the center 
of communication for the maintenance department. It is generally the only time when the 
entire maintenance group is assembled in one place. 

These meetings were observed approximately twice per week over the eight- 
month observation period, and changes in the communication process were observed 
during this time. In particular, the period at the end of the meeting after tasks had been 
turned over and discussed evolved into a forum for reflecting on QP issues, and raising 
concept challenges and concerns. The maintenance supervisor who facilitated the 
meetings became increasingly more adept at making the end of the shift meeting a time 
for individuals to be heard, and for reflection upon how recent situations were handled by 
the group as a team so that error was minimized. 


Overnight voicemail turnover. Another critical occurrence of communication was 
the turnover of work from night to morning shift. Where in larger commercial aviation 
companies such turnover is usually written, in Co.l overnight turnover is achieved via 
telephone voice messages. A special voice mailbox was created in the department for the 
sole purpose of passing voicemail turnover from night shift to morning (day) shift. All 
maintenance crew members, as well as pilots in the department, have access to that 
voicemail box at all times. The voicemails were left by any member of the maintenance 
night shift. Though no formal method of selecting the person to leave turnover was in 
place, the technicians generally attempted to rotate the person leaving the turnover. This 
was requested by the maintenance supervisor as a way to get various team members 
accustomed to the role. 

The turnover was accessed each morning by the morning technician on duty and 
the maintenance supervisor. The turnover message was then saved for approximately one 
day for access by anyone who might need to reference it. Of particular importance about 
the voicemail turnover system is that it was implemented largely as a result of attention to 
optimal communication spurred by the QP program. It represents the maintenance 
group’s awareness of the importance of bringing shared knowledge and information to 
bear on management's tasks. The maintenance supervisor made regular practice of 
reminding technicians to leave thorough turnover messages that consistently include all 
information necessary for the morning shift. 
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Challenging and validating concepts (Concept Alignment Process or CAP"), Much 
of the challenging and validating of concepts was observed to occur at the end of 
afternoon shift-change meetings. By creating a daily forum for the entire group to debrief 
at the end of shift meetings, a greater number of concepts were brought to the team’s 
attention. This in turn resulted in increasingly more challenges to concepts that required 
third source validation. Although the program intends for team members to actively use 
the process on the hangar floor and at all times, having a daily opportunity for the process 
to be practiced in the presence of the entire crew fostered used of the process elsewhere. 
Most importantly, it provided a daily opportunity to mitigate or avoid latent error in the 
hangar. 


Trust. Integral to the success of the program implemented at Co.l’s aviation 
department is trust. Because the premise of the concept alignment process is the sharing 
and validation of concepts, a culture in which open and assertive communication is 
valued and not punished is essential. For this reason, evidence of changes in trust was 
sought during crew observations and interviews. Survey results speak to this issue (cf., 
Figure 2), and field observations also showed evidence that trust improved as a result of 
improved communication processes. First, AMTs reported feeling increasingly more 
able to raise alternative concepts without fear of punishment. Though some technicians 
commented that they felt management would react negatively to raising certain issues, 
researchers here observed an increase in how vocal technicians were regarding safety. As 
QP became more deeply implemented, new and higher standards developed regarding the 
degree to which management was expected by the crew to hear and respond to 
challenges. Secondly, we also observed a decreased defensiveness regarding challenges. 
A major obstacle to implementation of a program of this type is the manner in which 
challenges to work and work processes are perceived by those who performed them. As 
the program evolved, we observed a change from defensive reactions to alternative 
concepts, to AMTs actually asking other team members for other solutions. That team 
members were actively seeking validation of their own work, both within themselves or 
from other team members, is an indication that many of them have come to trust their 
team members, management, and do not fear the consequences of having their work 
challenged. 


Casebook Development 

Among the products of the field research conducted at Co.l is a casebook in 
which instances of the use of the QP program between June 2002 and February 2003 
were documented by the researchers and returned to maintenance group and the aviation 
department manager for review. The casebook is included below. Some cases were 
directly observed and documented by the researchers during their twice-weekly visits and 
the other cases were gathered by the researchers in conversation with Co.l AMTs and 
their managers. Although this set of cases represents a large sample of QP application 
during this eight month period, all parties acknowledge that it is incomplete because 
some applications were inevitably forgotten, or overlooked, or otherwise considered 
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unsuitable for inclusion. The case book was a growing document with new cases being 
added regularly and old ones being edited. Technicians were always encouraged to read 
and discuss the cases, as well as add comments or changes to the cases in the book itself. 

The casebook served several functions in this research: a) it allowed for one 
method of documenting the research conducted at the facility, b) it allowed further data 
collection, as the comments and additions, by AMTs, in the casebook represented further 
data for better understanding and analysis of the culture, and c) the casebook acted as a 
tool toward change facilitation, as AMTs used the book to reflect upon their own 
behavior from a framework consistent with the program. The review, discussion and 
modification of these cases developed further understanding and use of the program by 
the participants. The cases represent a range of experiences with program 
implementation, from instances of successful error mitigation, to communication 
breakdowns, that offered learning opportunities for the crew. The cases are presented in 
chronological order. It will be noted that, in general, the earlier cases are less complex 
than the later ones and the former tended to be treated as less important for follow-up or 
closure. Through the feedback and reflective process, these stories were used by the crew 
to develop new methods of communication and, in some cases, new procedures for 
operation. 


A Casebook of Quantum-Pro Practices in a Corporate Aviation Maintenance Department 

ABOUT THIS CASEBOOK 

The following cases are observations of Quantum-Pro communication practices at 
Co. 1 . There are several purposes for our offering this casebook: 

1 ) Establish documented evidence of Quantum-Pro practices in the Co. 1 Hangar. 

2) Allow aviation professionals at Co. 1 to see and reflect upon their own use of 
Quantum-Pro communication principles. 

3) Provide the beginning of a forum for aviation maintenance personnel at Co. 1 to 
offer their own observations and stories that reflect the use of Quantum-Pro. 

Case 1 : Shortened Inspection Interval 
6 / 19/02 

Case of DeHaviland’s reduction to scheduled inspection intervals for the Twin 
Otter aircraft. After Co.Ts timely sign-off and subsequent rush several days later to fulfill 
the deferred inspection - and the next inspection was coming due - the Co. 1 
Maintenance Supervisor finally reached DeHaviland, who admitted that the short interval 
was in error. 
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Case 2: Task Card Translation 
7/12/02 

The case involved a translation problem from French to English in a progressive 
check document for the Falcon 50 aircraft. Task card called for removing "caps" from 
the horizontal and vertical stabilizers and lubricating their threads. All the technician 
could find were "plugs" filling holes in the control surfaces. He asked another technician 
about the "caps" and that tech thought they were those Tittle slotted plugs" that the first 
had already determined were the part in question. The tech brought it to the group as an 
example of CAP. 


Case 3: Landing Light Attachment 
7/12/02 

Apparently the bulbs have occasionally exploded landing lights on the Twin Otter 
aircraft. During a bulb replacement the broken lens and rubber gasket seemed to be 
glued onto the metal of the wing. Technicians present at shift change said that a tech 
(not present) had been known to attach the landing lights with silicone sealant - nobody 
present seemed to know why, so the concept couldn't be confirmed. Case was not 
further resolved. 


Case 4: Accidental Fire Alarm Activation 
7/12/02 

Two techs were working on one of the hangar doors. Not knowing the sequence 
of steps required to deactivate the fire alarm, they apparently removed the cap on the 
fire extinguisher as well as its reservoir, and set the alarm off. The techs asked the 
larger group, “Where should they have sought a third source of information?” 


Case 5: Pilot-Maintenance Briefing Coordination #1 
7/12/02 

The case dealt with advance communicating between flight crew and 
maintenance about the arrival time for pilots in the AM. A tech said he had asked the 
Flight Ops Manager about the concept of flight crews specifying in advance their ETA at 
the hangar in the morning. The F.O. Manager said he didn’t know about any such 
practice, and told the tech a good concept was to “call the crew to find out when they’re 
arriving - if he wants to know.” Case was not further resolved. 


Case 6: Pilot-Maintenance Briefing Coordination #2 
7/12/02 


Case also about flight crew lack of coordination with maintenance. This time it 
was advance guidance about which zones the aircraft should be parked for AM 
departure. Per the tech the pilots don’t know about the concept of parking or staging 
zones for a/c. Case was not further resolved. 
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Case 7: Pilot-Maintenance Briefing Coordination #3 
7/12/02 

A third lack of concept alignment with flight crews. Flight crews seem to have the 
concept that their own preflight briefing should precede the joint brief with maintenance - 
making maintenance techs wait around till they’re finished. Why then, reasoned a tech, 
don’t flight crews wait around for the joint debrief until maintenance has done their post- 
flight walk, 


Case 8: Fuel Filter Removal 
7/18/02 


The Maintenance Supervisor had been bothered by a Gulfstream document that 
specified that an in-line (EPA) fuel filter be removed from each engine, but didn’t call for 
any other parts to replace it. He had called and asked the manufacturer’s tech rep if 
there was a union or some other part they needed when the filter was removed. The rep 
had told him, no, but the associated injection pump would be replaced with a new type at 
the time the engine was rebuilt at mid-life. Still confused, two techs queried another tech 
recently trained at the Gulfstream school about the details. The third tech confirmed, 
and demonstrated, that the filter was easily removed without new unions and he had 
also heard that a modified pump (maybe with it’s own filter screen) would be installed at 
engine rebuild. 


Case 9: Validation of Twin Otter Power Data: Use of Proper Pilot's Operating 
Handbook 

7/25/02 


Maintenance were getting reports from flight crews on the Twin Otter aircraft that 
one engine wasn't making power. The next day, based on these reports, a technician 
increased fuel flow to increase torque. Even with the adjustment, the pilots still reported 
it not making performance. The tech asked the pilots, what are the other indicators they 
had, and never got an answer to this question. 

On Monday night (7/22), the tech again asked them about the other engine 
parameters. The flight crew didn't know about them, but then they flew it again and said 
it still didn't make power. The tech looked in the maintenance manual and the numbers 
were different from the one's the flight crew gave maintenance. The pilots were using a 
checklist supplied by a training vendor that was for training purposes only - not approved 
data. The tech asked the flight crew to get the proper operating handbook that indicated 
that the engine was making power. 

Another tech on duty said that "By talking through it with the pilots, we were able 
to understand where the problem was coming from." One of the pilots came in on his 
own time the next day to brief the chief pilot on the incident, so that he could pass it 
along to the other pilots to avoid the same error (using the improper data from the 
training version of the manual). 

In a shift meeting shortly after the incident : 

The Maintenance Supervisor tells the two techs they did a good job debriefing 
with the flight crew about Twin Otter performance. He pointed out that through the 
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debrief, they were able to validate that the crew was using a training document that was 
not approved for regular flight use and commended them for using the pilot’s handbook 
to validate and debrief with the pilots. 


Case 10: Landing Gear Fairing Door Bolt Case 
7/30/02 

The trunnion bearing on the Falcon’s landing gear would not take grease. The 
maintenance manual says if the bearing won’t take grease it must be replaced. Two 
technicians worked on the job on separate shifts. Tech 1 put the first landing gear 
cylinder retaining bolt on and tightened the nut to just less than specified torque (21 .39 
to 30.24 ft. lbs) so that the cotter pin was lined up with hole (see diagram and part). 
Unaware of this, Tech 2 attempted his side the next night and only got to about 10 lbs 
when the nut stripped. In the process of removing the stripped nut the bolt was 
damaged. He ordered a new bolt and 6 nuts, which were sent overnight. 

The Maintenance Supervisor learned of the stripped bolt by phone from tech 2, 
who mentioned it in his turnover call. The Maintenance Supervisor said that he didn’t 
disagree with the judgment to torque less than spec, but that he wanted to debrief it as a 
communication issue. He told Tech 1, “when you made that judgment, if you had offered 
to discuss it, the night shift might have avoided damaging a part. 

In a subsequent shift meeting the job was discussed and tech 2 asked tech 1 
how he kept from stripping the bolt. During the next meeting Tech 1 told the group that 
he torqued the bolt to just a fraction less than spec, and that he had fastened the bolt to 
just less than 20 lbs because that is where the cotter pin lined up. Tech 2 had used one 
of the new nuts and it torqued just fine. Tech 1 then re-torqued his side with a new nut 
and it torqued to spec also. Tech 2 suggested to that there should be something in the 
procedures that recommend a new nut. A third tech added that he thought an Engineer 
should probably be consulted if going outside of specs on this bolt. 


Case 11: Obtaining Written Validation After Talking to IVlfg. tech support 
8/6/02 

A technician was doing a detailed inspection on the left and right wheel wells of a 
Gulfstream aircraft. At the end of the inspection, the job card instructs to put protective 
coating on the wells as a renewal procedure. The tech wasn’t certain if that meant he 
needed to put it on now or wait - his hunch was that it wasn’t needed; but he had done it 
on another plane without knowing he didn't need to. The card read: "complete renewal 
to be done at next C check." The card came up before the next C check, so it turned out 
that the coating wasn't needed before the next C check. He called the Manufacturer’s 
tech support at the Maintenance Supervisor’s suggestion and they validated his 
interpretation of the manual. [In a later discussion between the Technician and the 
Maintenance Supervisor, they both agreed that it looked clear enough to them, but the 
Maintenance Supervisor had him call manufacturer’s tech support to be safe.] The tech 
asked the Maintenance Supervisor in the meeting if documentation would be needed 
from the manufacturer. In effect, the tech was questioning the validity of manufacturer's 
tech support's advice. The issue was discussed in the meeting in QP terms. Another 
tech added that further validation would be needed only if the first tech was not satisfied 
with the answer from manufacturer’s tech support. If he could not validate, then he 
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needed to go to judgment phase and choose the least risky (most conservative) action. 
It was finally decided by the group that documentation was not required in this case. 


Case 12: Crack Damage to Forward Bulkhead on Twin Otter aircraft 
8 / 21/02 


During routine inspection, a technician [1] found damage to the forward bulkhead 
on the Twin Otter. The Maintenance Supervisor took photos and sent them to 
DeHaviland, together with an email asking if it could be repaired in-house using the 
structural repair manual - DeHaviland said o.k. Tech-1 wrote up the discrepancy and 
showed it to another tech [2], Tech-2 made a proposal about tech-Ts approach to fixing 
the crack damage (cutting off front piece, inserting glass, aluminum, plastic as needed; 
driving aluminum tube into bulkhead to insert bolt). After stating his concept, tech-2 then 
invited the first tech to state a counter-concept. Tech-1 stated a potential counter- j 

concept, but then deferred to the Tech 2's initial approach and agreed that more 
discussion would be needed as the job progressed. Tech 2 later stressed the 
importance of obtaining as many perspectives as possible on an issue. He also noted 
that his statement of concept and the first tech's counter-concept are merely initial 
approaches, and dialogue must continue as the job develops. 

I 

I 

Case 13: Gulfstream Brake Line Dent Case 
8/22/02 

A tech was doing an early AM post-flight check from a late-night return and 
noticed a small dent in the brake line. When the Maintenance Supervisor came in the 
tech approached him about the dent and asked if it was small enough to be ignored. 

The Maintenance Supervisor looked in the manual for dent limitations and found that 
anything >0.010” required replacing the line. Since the dent was deeper than that, a 

replacement line was made and installed. ! 


Case 14: Hydraulic Filter Post-flight Squawk and Replacement 
8/28/02 

The hydraulic filter bypass indicator on a Gulfstream had "popped" (indicating 
possible need for maintenance). A technician saw this on the post-flight check, and 
removed and cleaned the filter. He replaced the filter and the popper did not go up 
(indicating ok). The tech approached another tech and debriefed the item with him - 
asked him what he thought and pointed out that the aircraft was going on a long trip to 
Asia. Tech 2 affirmed tech 1's concept that the filter should be changed anyway 
because the aircraft is going on a long trip. Tech 1 replaced the filter, even though the 
manual did not require this. Through the debriefing, a more conservative decision was 
made based on contextual circumstance that could not have been made by an individual 
following the manual. 
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Case 15: Nose Wheel Steering Failure 
8/28/02 


Nose wheel steering on a Falcon 50 failed on taxi into the hangar. This same 
problem has been occurring over the past year or two, and hasn't been fixed because 
maintenance cannot get it to fail in the hangar. In this instance, the squawk came in 
Tuesday night from the pilots to two techs. They tested it in the hangar, but it did not fail. 
The pilots were inclined to leave the squawk verbal, but one of the techs told them to 
write it in the discrepancy log because the problem has been occurring sporadically for 
some time. [Note: In the past, most pilots, even though they mention the issue on 
debrief, had dismissed it as a minor nuisance]. The squawk entered into the book read 
"nose wheel steering tiller sometimes disengages." The next morning, a tech picked up 
the discrepancy and looked at the nose wheel. He tested the tiller and said it was fine. 

In the shift meeting later that day, the Maintenance Supervisor asked the group 
about what they think should be done because the problem has persisted, but doesn't 
seem critical to the pilots. A tech commented in the shift meeting that as a mechanic he 
doesn't like the attitude of not knowing or caring what's wrong - others agree. The tech 
who picked up the squawk reports that he pulled the steering out and determined that it's 
50-50 as to what is causing the problem (tiller or amplifier).The Maintenance Supervisor 
says: "I want to do the right thing. The most conservative action according to Quantum 
Pro is to replace both tiller and amplifier." Another tech challenges both the 
Maintenance Supervisor and the first tech's concepts by saying that, even if we replace 
both, we still have no way of validating that it's fixed. All agree. 

No concrete decision was made during the meeting, but the dominant concept 
seems to be swapping the tiller with the other Falcon 50 in the fleet and seeing if it fails 
(because there is minimal risk on this item). This is eventually the course of action taken 
and the tiller was determined to be the problem. 


Case 16: Gulfstream Gross Landing Weight Case 
9/4/02 


On Thursday the Maintenance Supervisor asked Flight Ops Manager how much 
fuel to put in the Gulfstream for its trip on Saturday. The Flight Operations Manager said 
fill it up. The Maintenance Supervisor briefed Maintenance Technicians on Thursday to 
fill the plane. On Friday, pilots briefed with mechanics and said don't fill it up (reasons 
having to do with weather and other factors). Before take-off on Saturday, pilots entered 
cabin and closed door (pressurized plane). Realized they had an incorrect manual, and 
had to open the door and close it again. They took-off and then landed minutes later 
because the plane wasn't pressurizing (door not sealed). 

When Maintenance Supervisor learned about the plane returning, he asked 
about the pilots about the fuel load because he knew that the plane might be over the 
"gross landing weight." Pilot confirmed that they were not. Maintenance Supervisor 
asked the hypothetical question "What would happen if the plane did land over the gross 
landing weight?" Maintenance Supervisor and the pilot looked at the maintenance 
manual and it said that if there is no g-meter installed, then a special overweight landing 
inspection was required (this would have grounded the plane for a while, and taken man- 
hours). 
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Case 17: Twin Otter aircraft Start-Up “Generator Assist” 
9/12/02 


Two technicians have just recently returned from Twin Otter aircraft school. In 
the training, they learned the accepted start-up procedure, which differs from typical Co. 

1 practice. The Maintenance Supervisor said that he was taught in 1990 (his last Twin 
Otter aircraft training) to do a cross-generator start to keep the batteries from taking an 
extra hit (this is apparently no longer accepted). This prompted a third technician 
inspect the starter condition today. The Maintenance Supervisor said at the meeting that 
he intends to ask the check pilot why they are starting up this way, especially since it 
differs from the procedure in the Ops manual. Tech 3 added that he thinks it would save 
a lot of money in the long run (less wear on the engine and starter/generator). 

Research of starter/generator change history showed the starters were not going 
to the expected number of cycles. Expected starts are 1200 cycles. Co. 1 was 
averaging anywhere from 500 to 800 starts. These data were presented to the chief 
pilot, who subsequently issued a memo to all other pilots about a change in Co. 1 ’s 
SOP on the Twin Otter starting. 


Case 18: Falcon 50 Brake Change 
9/12/02 


Brakes had wear that were just out of spec, on a Falcon 50. Because the brake 
manual says to force test grip tubes when overhauling the brake, new grip tubes were 
ordered. However, they hadn’t arrived that day, and the aircraft was going on a round 
trip (2 landings) the next day to San Diego. 

At the shift meeting, the Maintenance Supervisor debriefed the item at the end. 
He said "I have an item I’d like to do a Quantum Pro debrief on and get your opinions.” 
The way he saw it, they could: 1) keep the brake installed with high confidence that it 
was fit for two more landings, or 2) replace the brake, but with the old grip tubes and 
replace the grip tubes after return from San Diego. The old grip tubes had been 
inspected, but under an old criteria that for some reason was no longer acceptable [need 
information]. 

A thorough discussion of the item followed during the meeting, and continued on 
the hangar floor as the techs decided what to do. The Maintenance Supervisor felt that 
the plane could go to San Diego and back with the brakes as they were, but checked his 
concept with the entire group and elicited a discussion, but not resolution. A number of 
technicians stood in the hangar after the shift meeting discussing the case when finally, 
one tech called a colleague at a company nearby who had an instrument that could 
adequately test the grip tubes. Another technician volunteered to take the grip tubes 
over to that nearby company for test. 

All grip tubes failed force test with a non-calibrated tester. Also, the T-bolt (1 of 
5, that indicated brake was worn beyond limits) was measured with micrometer and 
found to be 0.003”-0.004” shorter than other serviceable T-bolts. This T-bolt and spring 
cage assembly was replaced with serviceable items, brake wear was re-measured, and 
brake was found to be within limits. The aircraft continued in service and the brake 
assembly was replaced two landings later. 
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Case 19: Paint Stripping Case 
September 19-Early October 2002 

During the brake repair resulting from case # 18, a tech [1] disassembles the 
brake unit and turns the job over to the next shift. Maintenance Tech-2 finds the brake 
disassembled. Because normal procedure does not involve complete disassembly of 
the brake for overhaul, Tech-2 asks the Maintenance Supervisor why the procedure has 
changed, and what the new procedure is. Maintenance supervisor tells Tech-2 he 
doesn't have a ready answer to his question, and tells him to follow the Maintenance 
Manual, or wait for Tech-1 to return from out of town to debrief the incident. 

With the brake disassembled, a third Tech takes the task of doing visual 
inspection of the pads. The manual says that the stators on the brake pads must be 
visually inspected for wear. Tech-3 wasn’t comfortable doing a visual inspection without 
cleaning the brake first. Tech-3 made the decision on his own to strip the brake dust 
and rust off of the stators in the media blaster to clean the surface. In a subsequent 
meeting Tech-3 reports his action to the group. The Maintenance Supervisor tells me 
that he challenged Tech-3’s concept for a couple of reasons: glass bead could get into 
the cracks and cause problems, and the anti-corrosion paint on the stators is getting 
stripped off. The concept was validated by the group by looking at the brakes after a 
shift meeting and noting that no damage was done. In addition, the other Techs voiced 
opinions that Tech-3's action was acceptable. 

When Tech #1 returns, the crew debriefs at the shift meeting. Maintenance 
supervisor begins by raising the issue, and saying he wants to "open it up" to the group 
for debriefing. A discussion takes place in which Tech #1 explains that he had 
mistakenly taken the brake apart farther than normal shop procedure requires. Upon 
realizing this, he decides to go ahead and use the wet paint stripping method that is 
used when the brake is disassembled (as opposed to the dry method in which the brake 
does not need to be completely disassembled). The Maintenance Manual explains the 
wet and dry methods of paint stripping. 

The discussion turns to how much room for "professional judgment" there should 
be for deviating from normal shop procedure. A Tech raises the issue that the manual 
states that deviation from the procedure in the manual may be used as long as safety is 
not compromised and specs come out correctly. The intro section to the manual does 
give permission to deviate from manual procedure as long as safety is not compromised 
and specs come out correctly. 

The issue is discussed again the following Tuesday when a Tech who had been 
absent for the earlier meeting returns and makes a similar inquiry: Do we have a new 
shop procedure for brake overhaul? Tech #1 again explains his reasons for deviating 
from the normal procedure, and says that he plans to use the normal procedure in the 
future The brake was reassembled. 
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20. Loose Drag Brace 
Late December, 2002 

Some work was being done by 2 techs on the landing gear (changing out the 
nose gear piston because of corrosion) for a Gulfstream G5. They discovered "play" on 
the landing gear and left a turnover describing what they had found. The message 
described the problem and asked for validation of the looseness. 

The next day, the Maintenance. Supervisor asked a day shift tech to look at the 
problem. The tech validated that there was some play in the trunnion, and told the 
Maintenance Supervisor that the Maintenance Manual allowed for a certain amount of 
play. The supervisor called Gulfstream, who said that there have been some problems 
with this before. The day shift closed the loop by reporting the results of their validation. 
After the shift meeting, the 2 initial techs took the crew out to the hangar and show them 
the problem. As it turned out, the problem was in a slightly different area than the 
maintenance supervisor and day shift techs had understood, and the day shift saw the 
severity of the problem described by night shift. The supervisor then suggested the 
night techs remove the drag brace to look at it. This inspection revealed a loose 
bearing. After repeated measurements, on a phone call to Gulfstream engineering, the 
item was considered to be barely within limits. A letter from Gulfstream engineering 
followed, which allowed a larger size bearing to be installed along with the new torque 
value. 


The item was debriefed several days later using the QP debrief guide. In the 
process of the debrief, one of the techs suggested looking at the other G5 in the fleet to 
see if a similar problem existed. The maintenance supervisor agreed and had the night 
shift look at it. As it turned out, the same problem (play in the drag brace) existed on the 
other aircraft. The bore for the bearing was worn beyond limits. The bore had to be 
reamed and an oversized bearing installed. 

The case is an example of a latent problem that was detected because time was 
taken to properly debrief an item. Further, the maintenance supervisor alerted the field 
rep at Gulfstream about the problem. The rep said that he would write a report and that 
it would be distributed to all operators requiring inspection on the part within a specified 
time. 


21. OTTER “Hung Start” Case 
Second Week of December, 2002 

A "hung start" occurred in Twin Otter start-up on the #1 engine. The aircraft didn’t 
start within 30-seconds. The pilots were under the impression that there is a time limit 
for starting the Twin Otter. When the plane didn't start in 30 seconds, the pilot aborted 
the #1 engine start, started the other engine, and then came back to the engine that 
hung-up, which then started in time. Two concepts were raised during debrief at the 
shift meeting. One concept was raised that everything was in limits so no further action 
was needed. The only limit on start-ups for the Twin Otter is temperature, not time. 
Another concept was that the situation was unusual and should be watched closely. As 
a follow-up action, a tech began to record the number of seconds it took for the Twin 
Otter to start, and noted the conditions associated with longer start times (e g., sitting 
for a period of time). 
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It was found during a normal scheduled departure that the left engine of the Twin 
Otter had changed starting characteristics, and was noted as being sudden or non- 
progressive. A normal and typical start duration to a stabilized idle had been 
approximately 22 to 28 seconds, which had changed to 30 seconds or more, with a 
hesitation at approximately 30 percent engine speed (Ng). It was concerning to the 
crews who understood from their training there was a 30 second starter limitation. 

Upon discussion by the maintenance team using the maintenance manual it was 
determined the starter limitation was only regarding motoring the engine and not during 
a normal start, and that the only starting limitation was a requirement of 10 seconds 
maximum from "fuel on" to "light off'. This limitation allows for variations in operating 
environments such as field elevation, cold temperatures, etc. where a starter limitation 
could be exceeded. 

Even with this information passed on to the crews there was one occasion the start 
duration exceeded one pilot’s comfort level at approximately 35 seconds, where it was 
evident a hung start was occurring, and the start was aborted. It was discussed with 
the technician on the ramp and decided that the right engine would be started to 
recover the battery charge, then the left would be attempted again. The left engine 
started normally and the aircraft was dispatched. It should be noted that this start 
duration progressively decreased after each successive start during the day, and was 
typically only noticeable during the first start of the day. 

Discussion continued among the maintenance group regarding the change in starting 
duration. One concept was stated that although a change has occurred, there are no 
engine parameters or limitations being exceeded, therefore they really do not have a 
problem. A counter-concept was stated that regardless of this fact, this condition has 
not been normal for this aircraft, so something has changed and it should be 
investigated. 

It was agreed upon to look at the fuel nozzles of the engine, which had been 
serviced as the last maintenance action. Spare nozzles were installed on the engine 
and the set removed was inspected and flow checked with no discrepancies found. The 
left engine starting conditions remained the same. 

As validation for the counter- concept, a Pratt and Whitney tech rep was contacted 
for input, and recommended checking the start control valve. This component, when 
selected "on" during the start sequence, initiates primary fuel flow for light off, then a 
secondary fuel flow as Ng and fuel pressure increase for acceleration to idle. This 
secondary fuel flow normally begins at approximately 25 to 30 percent Ng, which 
correlated to the Ng noted in the start condition. There was a lot of debate concerning 
this component and the reported condition, but finally was agreed upon to replace it. 
The condition was not changed. At this time, yet another tech rep validated the original 
concept, stating that there are no limitations on the start duration. 

With this information in discussion by the maintenance team it was finally realized 
that they could only validate a concept by a third source, which we still had not been 
accomplished. Third sources, two individual engine tech reps, were not in agreement. A 
third tech rep was contacted who provided the Co.1 mechanics confidence that they 
were on the right path as far as troubleshooting, based on the information this was a 
shift from normal, and not an unacceptable condition. 

(2/26/03) As the aircraft has been away from Co.Ts facility since the end of December 
having other heavy maintenance accomplished, they were unable to continue working 
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on this condition. It was agreed -- and components procured -- to take the next 
sequential action before the aircraft is put back into flight status. 

The fuel control was changed (3/03). The engine condition subsequently became 
worse. After several days of working with another external vendor, it was decided that a 
“Min” flow adjustment was needed. This was the final fix for the affected engine. The 
removed fuel control was tagged “min flow adjustment needed” prior to install. 


22. Installation of Backup Attitude Indicator System in Twin Otter 
First week of February, 2003 

A request came from the Flight Ops manager for a back-up horizon (attitude) 
indicator system in the Twin Otter. The instrument was considered necessary so that 
the plane could be landed in the event of complete electrical failure. Maintenance staff 
considered two options: 1) install a pneumatic indicator that would feed from the engine, 
and 2) install a battery-powered indicator with its own dedicated battery. 

Two techs debriefed the item. Installation of this item came up recently from the 
Twin Otter inspectors in Temecula. Two techs were looking over the MMEL to see if the 
item was on the list. The concern raised was that if the item is installed and is not on the 
MMEL, then the plane could not fly if the part failed. The tech who received the request 
from the pilots raised a counter-concept that adding the indicator could make the aircraft 
more likely to be grounded if the part failed and was on the MMEL. The tech validated 
his concept by consulting relevant articles, looking at an advisory circular on Minimum 
equipment requirements for general aviation operations under part 91, and by conferring 
with other technicians. The tech learned from the circular, and validated from the other 
techs, that the part may be installed, and a petition made to a flight safety board to 
amend the MMEL (have the item taken off of the list). 

On Tuesday, 2/25 the maintenance supervisor reviewed the relevant articles and 
the advisory circular regarding requirements for maintenance MMELs and MELs. The 
maintenance supervisor expressed concern that there may be some aircraft components 
installed that need to be placed on an MEL, with maintenance and operations 
procedures included as required. The second option (installation of a battery-powered 
indicator) was completed by a third party vendor. The item was be placed on Co. 1 ’s 
individual operator MEL, and a spare will be put in stock. The maintenance supervisor 
said that the vendor would give them relief to fly the plane with the component installed. 
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Comparisons of Written Logbook Discrepancies Before and After QP Training 


The discrepancy log entries for two aircraft (a Twin Otter and a Falcon 50) were 
examined longitudinally across five-plus years following onset of the QP program for 
pilots and AMTs (originally in 1998), and the onset of the formal maintenance QP 
training in June 2001, and for 20 months thereafter. Because of the large volume of 
discrepancy log data, and the limited time available for data collection, only the first six 
months of each year from 1998 through 2002 (and January-March 2003) were examined. 
This method was used to provide consistency in comparison across years with the limited 
data sampled. 

The mean number of words contained in each discrepancy log entry was 
calculated for each year from 1999 through 2003. A longitudinal examination of 
discrepancy log entry practices by both pilots and maintenance is depicted in Figures 12 
and 13. Overall, the mean pilot discrepancy word count is higher than the AMT 
corrective action across the six sampled periods. However, the raw data results (Figure 
12) take an inverted-U shape, indicating an initial increase, a peak, and an ultimate 
decline. This effect is an artifact because the aircraft measured were out of service for 
long periods during 2000-2003 for airframe and avionics modification, as well as for 
heavy maintenance. 


Figure 12 


Error Mgt. Program & Written Communication 
(Uncorrected Analysis) 



— Pilots' Discrepancy 
Word Count 
— AMTs' Corrective 
Action Word Count 


In order to correct for this variation in aircraft service time, word counts were 
aggregated by month, and then divided by the number of flight hours for the respective 
aircraft in that month. Through this adjustment, an index was derived that controlled for 
the effect of heavy or light travel periods on log book entries. These corrected data are 
shown for each year in Figure 13. 
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Figure 13 shows the length of discrepancy text for the “subject aircraft” for 1998 
through the first quarter of 2003, corrected by flight hours. As shown in Figure 13, the 
distribution of mean number of words, divided by flight hours, arrayed across sampled 
months in each year remain higher following the re-introduction of the QP program in 
2000 and 2001. 

As with the uncorrected analyses, the pilot discrepancy entries have a higher word 
count average than the AMT corrective action entries. More striking, however, is the 
general shape of the curves in comparison to the unadjusted word count means. When 
corrected for flight hours, both pilot and AMT entries see a smaller decline or relapse in 
2003. The only significant increase in the word count index occurred between 1998 and 
2002 for the pilots [/(3 1 )= -2.53,/?= .017]. The pilot's wrote significantly longer 
discrepancy log entries in 2002 than in 1998. 


Figure 13 


Error Mgt. Program & Written Communication 

(Corrected Analysis) 



Pilots’ Discrepancy 
Word Count 


AMTs' Corrective 
Action Word Count 
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Summary and Conclusions 


This evaluative study of Co.l’s QuantumPro program employed a variety of 
research methods to gain an understanding of the culture on the maintenance side of a 
corporate aviation department. Survey measurement provided insight into key constructs 
such as trust and communication, observation and interviews illuminated the 
manifestation of these constructs in the specific culture under study, and the performance 
data yielded a concrete evaluation of behavior change as the department introduced and 
adopted the program. 

The study was unique to research in aviation maintenance human factors in that 
the focus was narrowed to the maintenance team. Observations revealed striking 
differences in communication practices among this sample as compared with larger 
commercial airline companies. Communication in this environment was more verbal, and 
as a consequence more personal and informal. The implementation of a new 
communication program in this department had positive measurable effects, as the team 
dynamics were such that the communication processes could be frequently and readily 
practiced. As measured by survey results, changes in behavior, attitude and general trust 
improved as team members gained experience with the new processes. Such results 
indicate a maintenance culture in which information is shared, alternative viewpoints are 
welcomed, and team members support each other on tasks. Further, behavioral impact on 
outcome performance was demonstrated with an examination of written discrepancies, a 
result consistent with earlier work by Taylor and Thomas (2003) in a large commercial 
airline. 

The aviation industry, particularly large aviation companies, may take interest in 
the dynamics of smaller team aviation cultures. The QP method at the current corporate 
facility engendered the kind of trust and information-sharing that will surely lead to 
mitigated errors or errors avoided in the long term. Increasing opportunities for one-on- 
one or small group briefings, along with a knowledge-sharing communication protocol, 
such as QuantumPro and CAP, should have significant impact on the attitudes, behaviors, 
and error rates of both large and small maintenance departments. 
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I. Evaluation of Behaviorally Based MRM Programs 

C. Company 2, Airline Maintenance Department - MEDA and HF Training 

• Pre- and Post-training surveys 

• Intentions 

• Follow-up survey & Interview Reports of Behavior 


Introduction 

The study in this section examined and evaluated an MRM training program at a 
large commercial aviation facility. Like Co.l, this particular program provides an 
example of one of the latest types MRM programs — behaviorally focused and expected 
to be maintained long after training is conducted. In addition to an examination of a 
behavior based MRM program, this Co. 2 study provides a good comparison of a 
behavioral HF program in a large commercial airline company to the much smaller 
corporate aviation department of Co. 1 . 


Method 

The Human Factors training program at Co. 2 had three primary purposes: (a) to 
create an awareness of selected human factors topics (communication, stresses, 
supervision, and disciplinary practices) that can cause maintenance errors or can cause 
errors to be concealed, (b) to familiarize participants with MEDA, the Maintenance Error 
Decision Aid (Allen & Rankin, 1995), together with the company’s intention to use 
MEDA, and (c) to familiarize participants with a less blame- and discipline-focused 
management system (Marx, 1998) and the company’s intention to apply it with respect to 
MEDA. 

Program Format 

Each 8-hour training session was limited to 25 people, and was conducted off-site 
between March and December 2002 for all maintenance department employees. 
Employees were made aware of the program through internal company bulletins and 
memos dispersed prior to training. MEDA is a process in which maintenance errors are 
reviewed and examined by means of a fixed set of human factors elements, to determine 
the cause of the error. This process relies on the candor of the AMT perpetrator in 
explaining his or her role in the error to a trained MEDA investigator/interviewer 
designated by company management — and, in the case of Co.2, by the mechanics’ trade 
union. The program implemented by Co.2 specified that no party would be punished for 
their participation in the MEDA process. Following the interview the investigator would 
prepare a report, which includes recommendations for changes to avoid similar errors in 
the future. 

Evaluation Format 
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Survey measurement. The Maintenance Resource Management Technical 
Operations Questionnaire (MRM/TOQ) was administered to training participants before 
the training and immediately afterward (Taylor & Thomas, In Press). The MRM/TOQ 
measured some participant background information together with their attitudes & 
opinions immediately before and after training. The post-training survey also measured 
participants’ feelings about the training and their intentions to change as a consequence. 
The MRM/TOQ was used again after six months to follow up all of those same items as 
well as to measure behaviors changed since the training. 

Interviews . Some 30 Aircraft Maintenance Technicians (AMTs) and first-line 
managers were interviewed in July 2002 and an additional 10 AMTs were interviewed in 
January 2003. They were asked about the experience in the HF/MEDA training and 
about the observations about its diffusion and subsequent implementation. 

Results 


Longitudinal Comparison of Attitude, Opinion, and Behavior 

Surveys . Of all (1,400) Co.2’s Maintenance Department employees who attended 
the one-day training session nearly 100% completed the survey immediately before and 
directly after the training. 

In the last three months of 2002 and the first two months of 2003, 600 
participants were surveyed again on the six months anniversary of their training. There 
were just over 600 employees who had completed the training between March and May. 
Some 215 of those 600 returned their completed questionnaires (a very good return rate 
of 35.8%). There is a high degree of comparability between the six-month follow-up 
survey and the larger sample from which it was drawn. It is important to note that intense 
maintenance labor contract negotiations were underway in Co.2 during late 2002 and 
early 2003. 

Table 2 and Figures 14 and 16 show basic background characteristics of the 
population at the time of the training. 

Table 3 shows the experience of the first 600 of those who attended training. 

Table 3 can be compared with table 2 to understand the differences between the sample 
of 600 and the entire maintenance department of Co.2. It is clear that those first 600 
differ somewhat from the whole department - being older and more experienced in other 
airlines, as well as having higher seniority at Co. 2. 

Table 3 can also be compared with Table 4, the experience of the 215 of the 600 
who returned the survey sent to them six months after training. Figures 15 and 17, which 
contain additional background characteristics of the six-month sample, show that 
mechanics are slightly underrepresented in the six-month sample, as is the base- 
maintenance section. Otherwise the six-month survey sample represents the first 600 
relatively well - they too are older, with more experience in other airlines and with Co.2. 
The six-month sample also appears to be better educated and has slightly more military 
experience. 
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Table 2 

Co.2 All Pre-training Surveys 



Years in 
Tech Ops 
at Co.2 

Years in 
military 

Years in 
trade 
school 

Years of 
college 

Years w / 
other 
airline 

Age 

N 

1033 

1149 

1135 

1126 

1126 

1385 

Mean 

8.871 

4.260 

.976 

1.449 

3.357 

42.73 

Median 

7.000 

.000 

.000 

1.000 

.000 

42.00 

Mode 3 

a 

.0 

.0 

.0 

.0 

45 

Std. Deviation 

7.4243 

6.5041 

1.2022 

1.7349 

6.4329 

10.446 

a Multiple modes exist. The smallest value is shown 


Table 3 

Co.2: First 600 Training Participants, March-May 2002 


Years in 

Years in 


Years w / 



Tech Ops Years in 

Trade 

Years of 

other 



at Co.2 military 

school 

college 

airline 

Age 

N 

588 499 

495 

484 

489 

582 

Mean 

9.404 4.206 

1.027 

1.444 

3.611 

43.75 

Median 

10.000 .000 

.000 

1.000 

.000 

44.00 

Mode 3 

2.0 .0 

.0 

.0 

.0 

44 

Std. Deviation 

7.5299 6.3999 

1.2403 

1.7807 

6.9299 

9.949 

a Multiple modes exist. The smallest value is shown 


Table 4 

Co.2 All 6-mo Follow-up Surveys 


Years in 


Years in 


Years w / 



Tech Ops 

Years in 

trade 

Years of 

other 



at Co.2 

military 

school 

college 

airline 

Age 

N 

203 

184 

179 

179 

179 

193 

Mean 

9.637 

4.761 

1.126 

1.813 

4.204 

44.38 

Median 

10.000 

.000 

1.000 

2.000 

1.000 

45.00 

Mode 3 

2.0 

.0 

.0 

.0 

.0 

37 

Std. Deviation 

7.3352 

7.2707 

1.1837 

2.0499 

6.6903 

9.634 


a Multiple modes exist. The smallest value is shown 
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Percent Percent 


Figure 14 

Co.2 Post-training Survey 


60 



Job title 

Figure 15 


Co.2 All 6-mo Follow-up 

50 



Job title 
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Percent Percent 


Figure 16 


Co. 2 Post-training Survey 

40 t 



Department 

Figure 17 

Co. 2 All 6-mo Follow-up 

50 t 



Line Mtc Other Material Services Engineering 

Base Mtc Shops Quality Control Planning 

Department 
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Attitude & Opinion Change Over Time 


Figure 18 



Supervisor Trust Value Coworker Assertiveness Effects of My Stress 
Trust and 
Communication 


II Pretraining (N=1,410) ■ Post-Training (N=1,389) n6-mo Follow-up (n=209) 


Figure 18 shows attitudes and opinions change after training and changed again 
six months later. 


Table 5 


ANOVA Co.2 Pre-Post-6Mo 



Sum 

Squar 

df 

Mean 

F 

Sig. 

SUPTRST5 

Between 

24.31 

2 

12.15 


.00 

Trust & Safety 

Within 

2149.91 

297 

.72 



5- 

Tota 

2174.22 

297 




TRUCOMCW 

Between j 

18.19 

2 


26.13 

.00 

Coworker 

Communication (5- 

Within 

Tota 

1040.28 

1058.48 

298 

299 

.34 



ASSERTV 

Between 

45.98 

2 

22.99 


.00 

Assertiveness (2- 

Within 

3581.95 

300 

1.19 




Tota 

3627.93 

300 


Hi 


EFFECSTR 

Between 

181.89 


90.94 

IKETifjTjV 

.00 

my Stress (3- 

Within 

1987.05 


.66 

1 1 



Tota 

2168.95 



■ ■ 



Table 5 shows these changes are statistically significant. Trust in supervisor’s 
safety practices rose slightly after training, but then decreased markedly six months after 
training. The value of trusting coworkers reverted to pre-training levels, while the 
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importance of stress effects remained higher than pre-training level. Only the value of 
assertiveness increased six months after training. This value of “speaking up” has been 
noted in earlier studies and explained as the result of frustration with the apparent lack of 
progress in the MEDA program in contrast with its apparent utility ( cf , Taylor, 1998). 

Figure 19 


Co.2 Scale Percentile Ranks 

(Comparison Populations of 18,000) 



and Communication 


■ Retraining (ISM ,41 0) □ Fbst-Training (N=1 ,389) BJ 6-mo Follow -up (n=209) 


Figure 19 shows Co2 attitudes and opinions to be low (or at best, about average) 
when compared with the larger aviation maintenance database representing participants 
of Human Factors training programs evaluated during the decade 1992-2002. 
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Figure 20 


Co.2 Enthusiasm 



Training can increase Training will be useful Training will change 
safety & teamwork to others my behavior (4-pt 

scale) 


□ Post-Training (N=1,389) B6-mo Follow-up (n=209) 


Figure 20 shows that enthusiasm for the program diminished after six months. 
Interviews conducted in Co.2 maintenance during early 2003 suggested that this is 
evidence for participant disappointment and discouragement in the MEDA program 
because it had not been as widely implemented as they had been led to expect. Table 6, 
below, shows the degree to which these differences between Post-training and six-month 
follow-up surveys are significant. 


Table 6 


ANOVA Co.2 Post-6Mo 



Sum 

Square 

df 

Mean 

F 

Sig 

TRNGEFEC This 

Between 

71.14 

1 

71.14 

160.48 

.00 

can increase safety 

Within 

700.41 

158 

.44 



teamwor 

Tota 

771.55 

158 




TRNGUSFL This 

Between 

79.82 

1 

79.82 

168.90 

.00 

will be useful to 

Within 

749.08 

158 

.47 




Tota 

828.91 

158 




TRNGCHBH This 

Between 

73.37 

1 

73.37 

102.48 

.00 

will change my 

Within 

1127.68 

157 

.71 




Tota 








1201.06 

157 
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Percentile Ranks 


Figure 21 shows that the post-training enthusiasm for Co.2 Human Factors 
program is normal or slightly above normal, but after six months it slips drastically into 
the lower third of all 18,000 participants surveyed 1992-2002. 

Figure 21 

Co.2 Total Enthusiasm Percentile Ranks 

100.0% 

83 . 3 % 

66.6% 

50 . 0 % 

33 . 3 % 

16 . 7 % 

0.0% 

This Training Can This Training Will Be This Training Will 
Increase Safety and Useful to Others Change My Behavior 
Teamwork 

■ Post-Training (N=1 ,389) □ 6-mo Follow-up (n=209) 
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Responses to Open-ended Questions About the Training 

Figure 22 


Co. 2 Post-training Surveys 



Good aspects of this training? 

Figure 22 shows the percent of participants, immediately after the training, who 
wrote one of a number of answers to the question, “What was the best aspect of the 
training?” Thus, post-training, fully one-quarter reported they liked “everything” and the 
next highest rating (15%) was for the MEDA process (“Error investigation”), followed by 
“good instructors.” Clearly the MEDA process is seen as a positive step. 
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Figure 23 


Co. 2 Post-training Surveys 



What would improve the training? 


Figure 23 shows the post-training answers to “What would improve the training?” 
Immediately after training, nearly one-half of the participants said the program needed 
nothing and was fine the way it was. Another 8% said either to make it longer or do it 
again as recurrent training. 
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Figure 24 


Co.2 All 6-mo Follow-up 



Good aspects of this training? 


Six months later, the predominant rating for “What was best?” (Figure 24) 
remained, “everything was good” (now down to about 15%). But also at six months, 
participants had a lower rating for MEDA (Error investigation), ranking it sixth instead of 
second as seen in Fig 22. “Good instructors” were still seen as third in order rated. By 
six months after their training, participants had not seen or heard much of the use of the 
MEDA process or seen systemic changes resulting from it. AMTs interviewed still 
viewed MEDA positively, but were skeptical that it would cause major improvements in 
lowered error rates. Some AMTs interviewed were more cynical in their assessment that 
maintenance management would not move to support the safety initiatives promised 
during the HF/MEDA training. 
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Responses to Open-ended Questions About Participants’ Expectations and Behavior 


Figure 25 


Co.2 Post-training Survey 










How will you use this training on the job? 


Figure 25 shows that immediately after the HF training Co. 2 participants said 
they planned to heighten their awareness, fight complacency, communicate more, and 
12% of the total said they expected to be involved in the MED A process (“Error 
investigation”). 
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Figure 26 


Co.2 All 6-mo Follow-up Surveys 

40 n 1 



What changes have you made because of the HF program? 

However, as shown in Fig 26, six months later, some 30% of those surveyed said 
that they NOT changed at all. Those who did report some change said that they had been 
less complacent and more aware of things around them. Ninth down the list of aspects 
changed (at less than 3%) was MEDA (“Error investigation”). During late 2002 and 
early 2003 only a few AMTs interviewed said they had seen evidence of positive changes 
caused by the MEDA process. On the positive side, some of those interviewed said that 
the promise, “that MEDA would not cause AMTs to be punished,” had been kept. 
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Figure 27 


Co.2 All 6-mo Follow-up 

20 7 



How will you use this training on the job? 


Also, at six months after training, 15% of participants said they didn’t expect to 
change further (Fig 27). At the same time 12% and 11 %, respectively, said they would 
be more aware or more thorough. Another 1 1% said they didn’t know what they would 
do. 
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Discussion 


These results for Co.2 show plainly how important it is for upper management to 
follow-through with commitments made (both implicit and explicit) in the HF training 
they cause their subordinate personnel to attend. The message of the training was clear 
and unequivocal: “Be prepared to communicate openly about maintenance errors (using 
MED A) and you will see us use that information to improve the system for the benefit of 
everyone.” 

In interviews, AMTs said that during the first eight or ten months of the new 
program they saw a few cases of MED A investigations (in all of which AMTs were 
treated fairly), but no further information about changes or recommendations for changes 
was noted by anyone. As a result, the enthusiasm for the program declined for both 
AMTs and their immediate management. In the end, little or no change in behaviors 
resulted from the training and, for the AMTs, future expectations to change declined. 

During the year in which the MED A training and program was studied, AMTs 
interviewed, frequently voiced a troubling paradox. This paradox referred to a company 
slogan stating that they would, “Be #1 in the industry in safety and compliance,” which 
contrasted with the paucity of safety-relation actions or corrections they observed, or 
heard about from others. This lack of action and communication, they felt, put in 
question, the brave words of the safety slogan. 

The MEDA program should have been 1) designed to maximize the number of 
valid investigations during the first year, 2) also designed to maximize the number of 
organizational and technical system changes guided by those MEDA cases, and 3) to 
widely communicate to AMTs and their supervisors the improvements brought about by 
MEDA cases and resulting recommendations. The first year’s lack of visible action and 
follow-though so thoroughly discouraged Co.2’s maintenance personnel, that the MRM 
program’s initial momentum may not be recoverable. 
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II. User-centered Analysis Tools 

One of the original intentions of the NASA-sponsored research into maintenance 
HF programs (Taylor & Robertson, 1995) was to provide the aviation maintenance 
industry a self-administered evaluation process. In order for a “do it yourself’ process to 
be effective it needs to include valid tools, proven techniques and a large experience base. 

The creation and testing of valid and reliable measurement tools have been 
underway for several years (Taylor, 2000b; Taylor & Thomas, 2003; Taylor & Thomas, 
in press), and the development of a user-oriented results calculator was reported as part of 
the research conducted in 2001 (Taylor, 2002). The experience base has been 
accumulating since the initial study (Taylor & Robertson, 1995) and has been used for 
comparison by percentile ranking since 2001 (Taylor, 2002). The advent of the internet 
provides the wide accessibility and thus permits the user-centered, self-administered HF 
program evaluation to become a reality. 

The heart of this process is the web-based survey, supplemented with an on-line 
results calculator. These elements and others are described in the present section. 

Web-based MRM Evaluation Survey 

As part of the evaluation program effort to give individual aviation facilities the 
ability to measure their own programs, we tested and implemented user-accessible on- 
line tools to quickly and easily obtain attitude and opinion data for MRM program 
evaluation. These tools are demonstrated on the public website: 

http://mrm.engr.scu.edu . 

These tools include on-line baseline and follow-up MRM/TOQ surveys, and on- 
line data-entry templates. 


1. On-line baseline and follow-up MRM/TOQ surveys. On-line survey measurement 
was made available for use before and after the implementation of MRM programs (e.g., 
MEDA, ASAP, Quantum-Pro). Participating maintenance organizations can direct their 
employees to the appropriate URL to complete a survey online. These data are then used 
obtain anonymous and confidential information about: 

a. The levels and importance of trust 

b. The value of assertive communication 

c. The value of stress management 

d. Enthusiasm for the program 

e. Readiness to change 

f. Specific intentions to change 

g. Selected variables about respondent background and experience 

Additionally, filling out surveys at the appropriate URL will give the maintenance 
employee individual feedback about where he or she scored in comparison to the larger 
aviation maintenance population. 


54 



On-line data-entry templates. On-line data entry templates were made available 
for use with paper and pencil pre-and post-training surveys. Typically, aviation training 
rooms are not equipped with internet terminals so it would be impossible to collect on- 
line survey data immediately before and immediately after a human factors training 
session. One answer is to continue to use paper and pencil surveys of the type found on 
the following websites: 

1 . for paper and pencil pretest: 

http://mrm.engr.scu.edu/NEW%20MRMTQ0-pre%20n 7item).pdf 

2. or paper and pencil post-test: 

http://mrm.engr.scu.edu/NEW%20MRMTOO-post%20t23%20itemTpdf 

3. for paper and pencil follow-up: 

http://mrm.engr.scu.edu/MRM-TOQ2003%206mo%20paper- 

pencil%20survev.pdf 


These paper and pencil surveys can be downloaded from the internet, printed out, 
and administered by the trainer; who then passes them to data entry clerks, who enter the 
data directly into the web-based database using the appropriate template. The templates 
are found at 

http ://mrm. engr. scu.edu/ quickenter.html 

Temporary clerical employees would have adequate skills and could be employed on an 
as-needed basis for the few hours required for this work. Paper and pencil surveys could 
also be used in conjunction with on-line surveys for those (hopefully few) respondents 
who do not have at-work access to the internet. Participating maintenance organizations 
(having entered the responses on these pre- and post-training surveys collected during 
formal MRM training sessions) would have immediate access to the descriptive results of 
the surveys using the on-line results calculator described next. 

On-line results calculator. As another product of our efforts to allow maintenance 
facilities to self-evaluate, an on-line results calculator was designed, and is demonstrated 
at website: 

htt p://mrm.engr.scu.edu 

Statistics from this calculator provide participating maintenance organizations 
include the following. Four reliable and valid attitude scales, three well-known measures 
of enthusiasm for the MRM program, and several background variables (e.g., age, 
education, experience). These measures and others are compared over time, and with the 
historical industry-wide MRM database. The calculator also permits filtering on a variety 
of background variables so that users can compare groups within their organization on 
the variety of measures made available in the system. For example, management at a 
particular facility might be interested in comparing night shift mechanics to day-shift 
mechanics, or more experienced employees to less experienced. The calculator allows 
those given access to the system (usually HF program managers and facilitators) to obtain 
this information on their own as they find it useful. 


55 



Web-Site Description (A list of technical terms follows at the conclusion of this section) 


The web site is a set of scripts and web pages to collect surveys (In this instance, 
to determine the success or failure of the Human Factors program in a company) and at a 
later time present the data in several ways. This is accomplished using an Apache server 
with Perl version 5. Apache and Perl are not new. Though a more technical description of 
the website is detailed below, the major components of the web site where the prototype 
exists are listed here: 

1) Web pages that present information and past publications. This is simply a web site 
that welcomes the user and invites them to try out the prototype as well as read 
previous papers on HF programs. 

2) Various web forms, some publicly accessible, others not directly linked that contain a 

survey. They contain the HTML versions of the pen-and-paper surveys that have been 
used before, and are simple uses of the HTML format. When submitted, the web 
browser submits the data into the form submission engine, described below. 

3) The form submission engine, a perl CGI script that takes the submitted data and puts it 

in the databases. Receiving the submitted data is done with the CGI module, and is 
not new. The data is then preprocessed, which consists of taking the information 
supplied in the form and adjusting it to match the expectations of the database, such 
as changing the birth year of "59" to be "1959". The program then saves this 
information to two tab-delimited databases: One is raw data, which contains all the 
submitted entries. The other is dependant on the company. The reason that a tab- 
delimited database was used was for ease of use: The database is human-readable, 
and writing an entry consists of writing a single line at the end of the text file 
(database). Once that the work is done, the form submission engine then directs the 
user, using the CGI module's ability to redirect, to the appropriate form for the results 
calculator. 

4) The report generator forms, like the templates of part 2, are simple uses of the HTML 

format. These forms allow the user to request demographics of the entries to be 
processed, including: 

a) The kind of survey (baseline, immediate follow-up, 2, 6, 12, or 18 month follow- 

up. 

b) The range of dates when the form was filled out and submitted, 

c) A range of various demographics filled out in the form, such as age, job title, 

shift, gender, department, years in the company, etc. 

An additional form for data analysts, has more items that can be requested, namely: 

d) The companies to use, 

e) Which open-ended questions to display a listing of the most commonly used 
words. 
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f) Which open-ended questions to display what key words were detected and where. 

g) A button to download the entries specified as a tab-delimited file for use with 

SPSS. 

h) Options to preview, in table format, the entries that would be downloaded by 
(6h), as well as which entries would be missed. 

When submitted, the web browser submits the data into the post-processing engine, 
described below. 

5) The post processing engine, also known as the postprocessor, report generator, or 
calculator, is a perl CGI script that presents the data stored in the database in various 
usable formats. Depending on which report generator form was used; the calculator 
chooses which database to use as well as which features to enable. Simple security 
measures based on the location of the files are used to determine if the form is the 
right one. 

Next, the postprocessor opens the database as a plain text file, and begins reading 
it. It uses a very simple form of caching to store up to 8 entries from the database at a 
time. A simple perl command is used to turn a single line of text into an entry composed 
of fields. For each entry, it updates the cache, reading a new entry. The oldest entry in the 
cache is taken to be processed. The processing first starts with comparing this entry to 
the 7 following, looking at the similarity of various responses to questions, to see if the 
user had sent in the survey, changed his mind, and sent it in again. If this is indeed a 
duplicate, this entry is discarded so to not repeat the later versions. Next, various fields 
are compared to lists, turning expected text into numbers, such as "Male" in the gender 
field becomes 1, "Female" becomes 2, and "decline comment" becomes 9. With open- 
ended questions, the first three keywords that show up closest to the beginning are used. 
It is these numbers that are compared with the requests of the form in order to determine 
if the entry meets the criteria of the search. Furthermore, composite entries, such as trust 
in supervisors, are made by averaging several entries. 

Next, if the entry is not a duplicate and meets the search request, its values are 
tallied for statistics, adding one to the count, the value to the sum, and the value squared 
to the sum of squares. The math behind the statistics are those found in any high school 
or college textbook, of the mean, count, and standard deviation from these values. 

With the statistician's form, the other items are computed. A total word count of 
various fields shows the most commonly used words and phrases. A display of the entries 
in various fields shows the number matched, and the location found in context. An ability 
to download a database to import into SPSS, a statistical program, is included. The 
database that SPSS would receive can be displayed for debugging reasons. 

Finally, after the last entry is read, the mean, standard deviation, and other well- 
known statistical analyses are performed, and the results are finally sent to the user for 
review, as well as a copy of the form they used to request, as to allow them to change 
selections and repeat the process. 
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A Brief Description of Technical Terms for the Web-based MRM Evaluation 
Survey and its Supplements. 

This document assumes average knowledge of the Internet and a familiarity with a 
typical web browser. 

1 . HTTP (Hypertext Transfer Protocol): 

1.1. Request for Comments 2616, the standard for transferring web pages from server to 
browser. 

2. Web server: 

2. 1 . A program that conforms to the server functions of HTTP (sense 1 ). 

2.2. A computer running a web server(sense 1 ). Web sites that are available via a web 
server(typically sense 2, although both senses are accurate) are said to be hosted by 
the web server. 

3. Web browser: 

3.1. A program that conforms to the client functions of HTTP(sense 1) and renders the web 
page. 

4. User: 

4.1 . A person using a web browser. 

5. Apache: 

5. 1 . A web server(sense 1 ) that is used by the prototype, (http://www.apache.org) 

6. PERL: 

6.1. Practical Extraction and Report Language, (http://www.perl.org/) a scripting language 
that the invention is written in. 

6.2. The interpreter of the PERL(sense 1) 

7. CGI (Common Gateway Interface): 

7.1 . A defacto standard of dynamically-made web pages, where a program or script is run, 
the resulting text sent back instead of a web page file. 

7.2. CGI program or script. A program or script that is designed to use the CGI(1) standard 

7.3. A PERL module (program library) that allows for the easy creation of PERL-based 
CGI (2) programs. 

8. cgi-bin: 

8.1 . a folder reserved for running CGI programs in. 

9. form: 

9.1 A web page that contains certain elements that someone at a web browser 
can input, and by pressing a button, can be submitted, or sent as input to a CGI 
program. An example would be a search engine website. The box where the search text 
is entered along with the button to start the search are part of a form. The results page is 
generated by a CGI program which has searched for the matches before presenting 
them. 
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III. 


The Evolution and Current State of MRM Programs 


• Definitions 

• Four Generations 

• Success 

• Implementation & Diffusion 

A. Definitions of MRM 

MRM has been broadly defined throughout the industry, and accordingly has been 
implemented in many different ways depending on definitions held by various program 
designers (Taylor, 2000a). We direct the reader to the following definitions of MRM as 
representative of where modem MRM programs are moving: 

■ . .an interactive process focused upon improving the opportunity for the 
maintenance technician to perform work more safely and effectively” 
(ATA, 2001). 

■ Part of MRM is training, but part of it is ongoing interactive error 
reduction 

■ It is an organizational and cultural change. 

As a result of over a decade of implementation, and a concentrated effort toward 
continuous evaluation, learning and program redesign, the nature of MRM has evolved 
from a one- or two-day awareness training course to a complete program that remains in 
implementation long a after training is completed. Further, modem programs strive for 
measurable, observable behavior changes rather than simply raising awareness. A 
framework for understanding this evolution is provided in the “four generations” model 
(Taylor & Patankar, 2001). 


B. Fourteen Years and Four Generations of MRM 

Communication skills training. The first maintenance programs to introduce the 
notions of interaction and safety dated from the late 1980s and early 1990s (Taggart, 
1990; Taylor & Robertson, 1995). They were patterned on flight operations’ Cockpit 
Resource Management training, in which the goal of improved communication skills and 
exchange of information was paramount. In that training participants were taught how to 
be proactive by speaking up effectively when they thought that doing so could avert an 
error, or an incident, or an accident. 

Human factors & safety awareness training. In the mid 1 990s the first Human 
Performance In Maintenance (HPIM) program training was developed and introduced by 
Transport Canada ( cf , Taylor & Christensen, 1998, pp. 144-147). Although HPIM 
training was inspired by the programs of the first generation, it emphasized “the dirty 
dozen,” a list of critical factors to be aware of in reducing maintenance errors. The 
designers’ intention was to initially introduce the list of 12 topics and to heighten 
awareness of three of those dirty dozen: Lack of communication, individual stress, and 
fatigue. Their intention was to later introduce additional training to cover the remaining 
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nine topics on their list and to begin to change mechanics’ behaviors. The initial phase of 
HPIM was widely used in North American aviation maintenance companies, but the later 
phases were not. As a result, widespread awareness of the dirty dozen was achieved, but 
without much action or behavior change. 

Error investigation and learning programs. _ Also in the mid 1990s maintenance 
learning from human error was pursued in several ways. A program called 
“Roundtables” (implemented for many months by the maintenance department of one 
large airline, cf, Taylor, 1994; Taylor & Christensen, 1998, 117-118, 180-181), and 
Boeing’s Maintenance Error Decision Aid (MED A) program (Allen & Rankin, 1995) 
were devised to learn from errors in the field through voluntary cooperation between 
AMTs and their management. In the late 1990s the Federal Aviation Administration 
encouraged further development of these early maintenance efforts with their Aviation 
Safety Action Program (ASAP), a program for both flight and maintenance operations 
based on voluntary collaboration between labor and management together with regulator 
mitigation of penalties for airmen’s’ violations (FAA, 1997). 

Error management and avoidance. In the late 1990s a program in maintenance 
was initiated to capture errors before they occurred (Patankar & Taylor, 1999). This 
program has continued to evolve in one aviation maintenance organization (Patankar, 
Taylor & Thomas, 2003) and it has emerged as a true exemplar of the proactive approach 
to reducing human error, unseen in the aviation maintenance industry since the 
introduction of communication skills training a decade earlier. 

Framework for the Four Generations 

These four generations of MRM interventions can conveniently be subdivided by 
whether they address maintenance personnel one at a time (“individuals”) or in social 
groupings or numbers greater than one (“groups”). The four types of interventions can 
also be subdivided by whether they advocate being reactive or proactive - the latter refers 
to taking steps to avoid an error or situation, while the former means responding to an 
event or situation after it has occurred. Table 7 portrays the two subdivisions for the four 
generations of interventions. 
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Table 7 


Four Generations of MRM Interventions 



Reactive 

Proactive 

Individual 

2. HF Awareness 
Training: 

■“Fight complacency” 
■“Fight stress & 
fatigue” 

1. Communication Skills 
Training: 

■ Active listening 

■ Assertiveness 


3. Error Investigation 

4. Error Management 

Group 

Process: 

Process: 

■MEDA 

■ Briefing & Debriefing 


■ASAP 

■ (CAP) Concept 
Alignment 


C. Comparing Sustained Success Among the Four Generation Models 

In the section to follow, the four models will be compared on five criteria. 

• Purpose of the program as described by the developer/implementer 

• Resulting attitude changes by participants 

• Specific intentions to change of participants 

• Resulting behavior changes by participants 

• Changes in safety performance in the organization 
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MRM Generation 1: 1989-94 (summarized from Taylor & Robertson, 1995). The 
purpose of the training was described as providing communication skills to improve safety 
and efficiency. Once begun, enthusiasm for the program was high. Some 80 to 90% of the 
participants in the training said they liked it. Comparison with the results to the same 
questions from flight operations groups, measured earlier by others, showed mechanics 
reacted even more favorably. 

The changes participants in this first generation program intended, were at first 
rather passive, but they became more active as time went on. Immediately after training only 
10% said that they would more actively communicate. Two months and 12 months later 
25% 38%, respectively, say they expected to more actively communicate - in particular that 
they would be more assertive in their dealings with others. Twelve months after the training 
over 40% say they did communicate more actively. 

Safety performance also improved. In particular aircraft damage & lost-time 
injuries declined for two years after training. These changes in performance were also found 
to be related to the training. For instance, significant correlations were found between higher 
assertiveness attitudes and lower subsequent occupational injuries. Once the program 
concluded and its promotion by management ceased, the signs of success diminished by 30 
months after its onset. 


MRM Generation 2: 1996-99 (summarized from Taylor, 1998, 2000c). The 
purpose as stated in the introduction of this type of training was to create an awareness of 
human performance on errors and personal safety. Enthusiasm for change was typically very 
high. Immediately after training over 90% mechanics in four airlines were positive about 
their programs and nearly 95% said they expected to change in some way. Those expected 
ways of changing were largely passive. More than 30% said they would fight their own 
complacency and be more aware of their stressors and stress effects. Only about 15% expect 
they would actively communicate. 

What changes took place? Two and six months after training about 25% said they 
were less complacent and had become more aware of themselves and their stressors. Only 
about 5% said they had been more actively communicating two and six months after training. 
In addition, over 30% said they had not changed at all two and six months after training. 

After six months over 25% said the program would not cause them to change. 
Participant intention to further communicate more actively drops to less than 10%. Many of 
the respondents to the six and twelve month follow-up surveys blamed the lack of 
management follow-through on their apathy. They had expected, they said, to see their 
managers and supervisors communicating in new ways, and emphasizing safety more 
vigorously - but this hadn’t happened. Raising trainees’ expectations about changes but 
without much management follow-though had the effect of discouraging and frustrating 
many (at least a third) of mechanic participants, who said they would not support the 
program or try to implement it in their workplace. 

There were measurable safety effects in several of the programs. Aircraft damage 
improved for about one year after the training was completed. But without further 


62 



encouragement from management, the improved safety trends diminished. During a period 
following training, significant correlations were found between attitudes about stress 
management and subsequent safety performance. 

The outcome of generation 2 programs was not altogether successful. In two of 
the airlines we studied between 1998 and 2001 the programs terminated before completion. 
Although our report for 2000 (Taylor, 2000c) reported the success for generation 2 MRM 
programs in large airlines, that same report also noted the less successful experience of one 
airline (c/, “Co.E”). The other, ultimately unsuccessful generation 2 program, was labeled 
“Co.G” in that report, and its program had not commenced at that time. 

Our report for 2000 also revealed that many of both Co.E and Co.G’s mechanics 
had been hired with past experience in more than one airline. Patankar (1999) observed that 
when maintenance personnel with a high level of experience from one organization are 
relocated to another organization, sometimes, during periods of organizational turmoil (e.g., 
reductions in workforce, contract negotiations, corporate mergers/acquisitions) they tend to 
locate fellow expatriates in the new organization and compare the past with the present, often 
at the expense of their present employer. He called this pattern of past experience “sub- 
organizational mosaic,” or SOM. This SOM construct was subsequently empirically tested 
and confirmed (Patankar & Taylor, 2000) 

In both Co.E and Co.G the effect of sub-organizational mosaic was heightened by 
labor contract negotiations during the time the MRM training was undertaken. These two 
airlines also experienced the maximum effect by initially including in the training, stations 
where mechanics had unusually high multi-company experience (Taylor, 2000c). The result 
in both companies was resentment that effort and cost was being expended for human factors 
training during the time when protracted and heated negotiations over wages were taking 
place. In Co.E the high-SOM stations had relatively low levels of enthusiasm for change; 
with only 50%-70% of mechanics positive about the program, and the same proportion who 
expected to change their behavior. Likewise in Co.G just 70-85% felt positive about their 
MRM program with 65% who said they expected to change as a result of program. The sub- 
organizational mosaic affected the mechanics tolerance for their employers’ safety programs 
and caused them to vocalize their demand that the programs be discontinued. Both Co. E and 
Co. G discontinued their MRM programs before any performance results could be 
determined. 


MRM Generation 3: 1993-2002. Early attempts to reduce maintenance errors by 
discussing resulting incidents with mechanics, and then making changes to prevent future 
occurrences, were successful in making important process improvements (Taylor, 1994; 
Allen & Rankin, 1995; Taylor & Christensen, 1998). Until recently these efforts to 
investigate incidents and improve process had little or no publicity and were typically 
implemented by quality assurance managers who conducted the programs without active 
participation by maintenance employees. 

With FAA participation and encouragement the ASAP approach has been 
implemented in the maintenance departments of several U.S.-based Part 121 airlines during 
the past two years. Planning for independent evaluation of these ASAP programs has only 
just begun and it will be several years before public reports can be expected. Although a 
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condition of these programs is an internal evaluation and report of the first 18 months 
experience, none of these reports have become publicly available. 

On the other hand the MEDA program has recently been incorporated into larger 
human factors efforts, which have been independently evaluated. The “Co.2” activities 
described and evaluated in section I-C of the present report is an example of such a MEDA- 
based program. The evaluation has taken place as an entire maintenance organization 
undertook maintenance human factors training similar to the Generation 2 HPIM program, 
except that it included considerable material on the theory and practice of MEDA, as well as 
suggestions on how to modify the company’s disciplinary practices and policy (cf, Marx, 
1998) in order to improve trust and understanding between management and labor. It is clear 
that employee trust for management’s support of any Generation 3 program is important. 
Measures of trust were available for the evaluation and can be compared over time. 

The following comparative assessment is based on the “Co.2” activities evaluated 
in section I-C of the present report. It’s implied purpose was to provide maintenance 
department employees with 1) an overview of the intent and workings of the MEDA process, 
2) description and justification for a “blame-free” discipline policy that encourages 
participation in the MEDA process, and 3) an introduction to some of the human factors that 
cause errors and mistakes. 

Changed attitudes included initially high enthusiasm for the program. For 
example, immediately after training over 95% liked it and thought it would be useful. 
However, six months later this endorsement dropped to between 75-80%. Six months after 
training, trust in one’s supervisor’s safety practices also decreased markedly from post- 
training levels. The value of trusting coworkers also reverted to pre-training levels (cf.. Fig. 
18). In Section I-C, Co.2 was seen to have initially trained those AMTs with extensive 
experience in other airlines (cf, Tables 2-4). These initial trainees fit the conditions for sub- 
organizational mosaic, or SOM (Patankar, 1999). As the training’s vanguard, these high- 
SOM individuals were responsible for communicating their initial impressions (and thus the 
impression also of Co.2) of the MEDA program. Equally important, this same group was 
asked six months later (not incidentally during a time of heated contract negotiations) to 
verbalize their subsequent impression of the program. Their frustration and discouragement 
with the pace of the program, combined with the invidious comparison of Co.2 with their 
former employers, acted to produce the negative views evident in Figures 21, 22, 26 and 27 
above. 

Intentions to change immediately after training (cf, Fig. 25 above) showed that 
about 12% say they planned to apply the MEDA error investigation process (this included 
about 28% of all the managers surveyed). Another 28% of the total participants said they 
would fight their own complacency and be more aware of their stressors. A further 25% said 
they would actively communicate. 

What were their changes in behavior? Figure 26 showed that six months after 
training about 30% of all participants (including about 20% of all the managers) said they 
have not changed at all, and nearly the same proportion (cf, Figure 27) think the program 
would not change them in any way. A small proportion (less than 5% total) of all Co.2 
respondents said they had used the MEDA process or experienced the “blame-free” culture. 
Some 15% of all managers said they had used these techniques. It is interesting to note that 
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six months later over a third of the managers said they would use the MEDA process or the 
“blame-free” culture. 

Safety results were not available from Co.2 by the first quarter of 2003. We were 
told that there were some 30 MEDA investigations conducted during the first year of the 
program, but it was also reported that few, if any, of the recommendations included in those 
reports had been implemented. 


MRM Generation 4: 1995-2003 (based on Patankar & Taylor, 1999 and on the 
research reported in section I-B above). Between 1995 and 1998 an early attempt to avoid 
and manage maintenance errors by modifying the employee decision making process, was 
successful in Co. 1. Between 1998 and 2001 there were many changes in that company’s 
aviation maintenance department (including, but not limited to downsizing personnel, 
liquidating one-half of the fleet, corporate mergers and acquisitions, rapid department growth 
requiring rehiring some former employees and many new ones, and finally the purchase of 
several new aircraft). In 2001 this program (now officially called the “QuantumPro 
Process”) received added management support and endorsement compared to the prior 
period. After a two-day training course, a subsequent six months of individual tutored study 
in the process, and a final six months of attempts to implement the decision making model, 
the members of the department requested us to interview everyone about the program and 
provide them with our findings, our conclusions, and our recommendations for next steps. 

Following interviews and observation during the first half of 2002 the following 
summary was presented to the aviation department by the present researchers. 

“•PLUS: The QP structure is helpful in standardizing routine communication 

-Better inter-shift communication now than a year ago. 

-Briefing checklist used on regular basis. 

•MINUS: Local modifications dropped the daily affirmation of QP and the assignment of 
roles & responsibility 

-Some complain about loose roles and responsibilities for invoking the QP in 

non-routine situations 

•MINUS: The definitions of the “5Ps” (purpose, philosophy, policies, procedures, 
practice) are problematic. 

-5Ps for the Company are still unclear. 

•MINUS: Little proactive use of the QP process by management is noted” 

These were our conclusions: 

“•Your company’s more trusting workforce should be an advantage in implementing the 
QP program 

-But progress is very slow and may die at this point. 

•Post-training attitudes improved 

-And improved again in the follow-up, but your attitudes are not yet behavior. 
•Your modifications of the program have reduced its scope of implementation and the 
opportunity to practice unfamiliar or uncomfortable behaviors. 

•The program can be strengthened with more active and consistent management 
participation.” 
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These were our recommendations: 

•’’Collaborators in a program like QP will have different points of view. 

•When these different points of view occur they must be aired fully and frankly. 

•Once a final decision about the use of the process is reached it is the duty of all 

collaborators to carry out that decision. 

•The program can only move forward with full and frank communication, starting now.” 

The application of QuantumPro Process (QP) thereafter began in earnest during 
the last half of 2002. 

This is the comparative assessment for “Co.l” activities described and evaluated 
in the section I-B of the present report. 

The purpose of the QP training was to provide Co.l maintenance department 
employees with skill, ability and organizational sanction to apply the QuantumPro process. 
This process was intended to eliminate current errors and prevent future ones from the same 
systemic source. Attitude surveys showed initially high enthusiasm for the program. 
Immediately after training over 90% like QP and thought it would be useful. Twelve months 
later this endorsement dropped to about 80% & remained so at 18 months after training as 
well. Twelve & eighteen months after training, trust in one’s supervisor’s safety practices 
increased markedly from post-training levels (Figure 2, above), as did the value of 
assertiveness (Figure 4). 

Immediately after training 2/3 of the participants say they intend to change as a 
result of the program, but less than 25% said they would use QP. By twelve months after 
training the intention to change has risen to nearly 90% and remained at that level at 18 
months. At that time one third specifically say they would apply QP and a further 50% say 
they expected to more actively communicate. Participants increasingly reported using 
“CAP,” (the simple, repeatable communication/decision protocol and unbiased, third-party 
validation of concepts) as a means to resolve conflicts. 

Changes in behavior were observed occasionally during the first year after 
training, and thereafter observation became more frequent and regular. We observed an 
increased use of the process and discussion about it during the year and a half following 
training. Twelve months after training 14% of participants reported that they had applied 
QP. Over a quarter said they were actively communicating. By 18 months after training 
over a third said they had applied QP and another quarter reported that they are actively 
communicating. We noticed improved communication, both written (logbook), and verbal 
(shift turnover). 

Safety Performance improved as a function of error reduction: Over the course of 
Co.l’s most recent experience, beginning in June 2002, 22 QP cases were documented in 
maintenance. With eight mechanics in Co.l’s maintenance department, that eight month 
record represents an annual average of 3+ errors avoided per mechanic during this most 
recent period. 
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III-B. State of MRM, Conclusions: 

1 . Open communication for safety “works.” Generation 1 ’s communication skills 
training, and its focus on individual commitment to communication and safety was 
successful - as long as top management continued to visibly support the program. 

2. Individual awareness “works” (but it’s not communication). Generation 2 emphasis 
on individual values and safety was largely successful - but if management support 
for continuing and expanding the effort was not visibly noted, the program results 
reversed and mechanic frustration and discouragement resulted. The negative effects 
of organizational turmoil on the continuation of such programs is noted - especially 
when the mechanics involved have much past experience with other employers. 

3 . Error investigation programs (Generation 3) are enthusiastically greeted by 
maintenance personnel, but unless the program is seen to diffuse and have impact on 
system safety it rapidly looses favor; and trust of management, and interest in such 
programs diminish. Such a program may not improve communication, nor is the 
program, by itself, intended to do so. 

4. Generation 4 focuses on proactive error management and is based on the use of 
decision making processes requiring open communication. It is a program based on 
behavior change (work related interaction among all parties) for error avoidance. The 
example evaluated here shows that despite some mechanic discomfort with the new 
“openness,” the change throughout system has progressed over the months following 
its re-introduction. Trust in management and the value of assertiveness have both 
improved over the period. 

In general, we may conclude that Generation 4’ s joint focus on strategy, structure, 
and leader support, combine with open communication, individual awareness, and safety 
processes for the highest potential for improving safety performance observed by these 
investigators, since 1989. 
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IV. Recommendations 

Taylor and Christensen (1998) offer evidence that many MRM programs did not 
reach their full potential due to lack of management follow-up. Studies have shown that 
many aviation maintenance managers experience high mobility, and therefore they may 
not have the extended time required to consistently support the MRM programs they find 
themselves involved in (Patankar & Taylor, 2000). It has become clear that MRM 
programs should be integrated with the core organizational purpose if they are expected 
to survive more than a few years. 

Patankar & Taylor (2000) offer one approach toward making MRM programs 
independent of the changes in management and the effects of mechanics’ past 
organizational experience. That approach is to have a documented and integrated human 
resources master plan which includes maintenance and which is approved by the 
President/CEO of the airline. This plan should clearly identify the anticipated outcomes 
of the program, associated time-line, and budget. It should be results-driven. All 
management personnel should then be held accountable for abiding by the plan. Unless 
all management personnel are evaluated for their implementation of the master plan, the 
human factors program may not get consistent support when the management changes. 
Management should mandate the behaviors and goal attainment of MRM programs. 
Employees appreciate the MRM programs once they have been involved and their initial 
enthusiasm and favorable attitudes are literally universal. They want to believe in the 
MRM message, but need guidance and leadership to behave effectively. Data show that 
management appointments change every five to six years, so it would not be prudent to 
think that a certain favorable manager would be able to support a MRM program forever. 
Program champions are essential to initiate the training and implementation, but these 
champions must also make sincere attempts to make the program independent of 
themselves. 

Almost a decade ago we spoke about “three pillars” of successful culture change 
in aviation maintenance (Taylor & Robertson, 1994). Those three pillars were, 1) 
unequivocal top management support and vision of the purpose for the change; 2) a well- 
conceived and relevant intervention, for behavioral change; and 3), timely, appropriate 
feedback through a broad range of measurement and evaluation activities. 

This final report continues to add evidence for the truth of these three pillars. 
They continue to be valid. The contribution this present research makes is that it 
documents a relevant and effective behavioral intervention, as well as providing self- 
administered measurement tools for appropriate feedback. 
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